python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 4163826 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 21>10 please execute split_portfolio.py -i 20699412 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2601838'] with mtr_portfolio_ids : ['20699412'] and first list_photo_ids : [] new path : /proc/4163826/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.1905696392059326 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 21 00:14:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3183 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-21 00:14:33.759219: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-21 00:14:33.787128: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-21 00:14:33.789360: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f00a8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-21 00:14:33.789417: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-21 00:14:33.793625: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-21 00:14:34.040237: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2445ea60 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-21 00:14:34.040294: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-21 00:14:34.041427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-21 00:14:34.041842: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-21 00:14:34.045054: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-21 00:14:34.048147: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-21 00:14:34.048543: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-21 00:14:34.051582: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-21 00:14:34.053024: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-21 00:14:34.058099: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-21 00:14:34.059106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-21 00:14:34.059182: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-21 00:14:34.059715: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-21 00:14:34.059731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-21 00:14:34.059741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-21 00:14:34.060645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2731 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-21 00:14:34.332034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-21 00:14:34.332149: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-21 00:14:34.332172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-21 00:14:34.332192: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-21 00:14:34.332207: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-21 00:14:34.332222: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-21 00:14:34.332236: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-21 00:14:34.332252: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-21 00:14:34.333079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-21 00:14:34.333839: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-21 00:14:34.333868: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-21 00:14:34.333884: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-21 00:14:34.333898: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-21 00:14:34.333913: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-21 00:14:34.333927: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-21 00:14:34.333944: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-21 00:14:34.333960: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-21 00:14:34.334777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-21 00:14:34.334802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-21 00:14:34.334811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-21 00:14:34.334819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-21 00:14:34.335714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2731 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-21 00:14:45.189263: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-21 00:14:45.367112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-21 00:14:46.629943: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.630027: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-21 00:14:46.630597: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.630638: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-21 00:14:46.637600: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.637619: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-21 00:14:46.638129: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.638144: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-21 00:14:46.644571: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.644590: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-21 00:14:46.645098: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.645113: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-21 00:14:46.675793: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.675849: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-21 00:14:46.676374: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.676403: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-21 00:14:46.682281: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.682304: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-21 00:14:46.682868: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.682884: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-21 00:14:46.716471: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.717000: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.718795: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.719331: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.762861: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.763500: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.765651: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.766164: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.796190: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.796710: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.798355: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.798869: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.805314: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.805886: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.807761: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.808299: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.814889: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.815447: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.817099: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.817615: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.849053: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.849608: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.850133: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.850656: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.854658: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.855200: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.873949: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.874480: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.875065: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.875629: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.888226: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.888794: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.889355: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.889914: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.894292: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.894864: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.899522: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.900102: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.912424: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.913027: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.917200: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.917768: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.918328: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.918886: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.919741: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.920311: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.930770: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.931397: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.931992: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.932563: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.933152: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.933720: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.948971: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:46.949505: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.012137: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.012697: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.029126: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.029735: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.030313: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.030881: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.035023: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.035561: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.036078: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.036593: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.037523: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.037545: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-02-21 00:14:47.047826: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.048346: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.057225: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.057747: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.058272: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.058787: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.059355: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-21 00:14:47.059964: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.17G (2328756224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.08586 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 31 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 13 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 17 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 17 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 41 Detection mask done ! Trying to reset tf kernel 4164438 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1769 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2962 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.003813505172729492 nb_pixel_total : 123789 time to create 1 rle with old method : 0.1440718173980713 length of segment : 460 time for calcul the mask position with numpy : 0.00044989585876464844 nb_pixel_total : 16251 time to create 1 rle with old method : 0.020827531814575195 length of segment : 173 time for calcul the mask position with numpy : 0.00011444091796875 nb_pixel_total : 2482 time to create 1 rle with old method : 0.0034248828887939453 length of segment : 53 time for calcul the mask position with numpy : 0.0017194747924804688 nb_pixel_total : 67792 time to create 1 rle with old method : 0.09400773048400879 length of segment : 416 time for calcul the mask position with numpy : 0.4023568630218506 nb_pixel_total : 332227 time to create 1 rle with new method : 0.02210831642150879 length of segment : 777 time for calcul the mask position with numpy : 0.00029778480529785156 nb_pixel_total : 10263 time to create 1 rle with old method : 0.011616706848144531 length of segment : 119 time for calcul the mask position with numpy : 0.0007402896881103516 nb_pixel_total : 36527 time to create 1 rle with old method : 0.04126095771789551 length of segment : 214 time for calcul the mask position with numpy : 0.005917072296142578 nb_pixel_total : 320948 time to create 1 rle with new method : 0.1034078598022461 length of segment : 775 time for calcul the mask position with numpy : 0.0005145072937011719 nb_pixel_total : 15863 time to create 1 rle with old method : 0.018912315368652344 length of segment : 145 time for calcul the mask position with numpy : 7.367134094238281e-05 nb_pixel_total : 1064 time to create 1 rle with old method : 0.001430511474609375 length of segment : 30 time for calcul the mask position with numpy : 0.0001895427703857422 nb_pixel_total : 7442 time to create 1 rle with old method : 0.00853109359741211 length of segment : 181 time for calcul the mask position with numpy : 0.00017690658569335938 nb_pixel_total : 5627 time to create 1 rle with old method : 0.007037162780761719 length of segment : 98 time for calcul the mask position with numpy : 0.00020265579223632812 nb_pixel_total : 8430 time to create 1 rle with old method : 0.009986639022827148 length of segment : 95 time for calcul the mask position with numpy : 0.0008869171142578125 nb_pixel_total : 51362 time to create 1 rle with old method : 0.05767107009887695 length of segment : 348 time for calcul the mask position with numpy : 0.0013053417205810547 nb_pixel_total : 65967 time to create 1 rle with old method : 0.07546114921569824 length of segment : 364 time for calcul the mask position with numpy : 0.00031304359436035156 nb_pixel_total : 7707 time to create 1 rle with old method : 0.009024620056152344 length of segment : 140 time for calcul the mask position with numpy : 0.0017693042755126953 nb_pixel_total : 109916 time to create 1 rle with old method : 0.12327790260314941 length of segment : 526 time for calcul the mask position with numpy : 0.00041413307189941406 nb_pixel_total : 17000 time to create 1 rle with old method : 0.019522905349731445 length of segment : 116 time for calcul the mask position with numpy : 0.00038552284240722656 nb_pixel_total : 20786 time to create 1 rle with old method : 0.027527809143066406 length of segment : 182 time for calcul the mask position with numpy : 0.0004055500030517578 nb_pixel_total : 5374 time to create 1 rle with old method : 0.006600379943847656 length of segment : 107 time for calcul the mask position with numpy : 0.0008351802825927734 nb_pixel_total : 15364 time to create 1 rle with old method : 0.019974708557128906 length of segment : 254 time for calcul the mask position with numpy : 0.002905607223510742 nb_pixel_total : 72224 time to create 1 rle with old method : 0.08039140701293945 length of segment : 452 time for calcul the mask position with numpy : 0.004110574722290039 nb_pixel_total : 176405 time to create 1 rle with new method : 0.009091615676879883 length of segment : 583 time for calcul the mask position with numpy : 0.0008544921875 nb_pixel_total : 13588 time to create 1 rle with old method : 0.015556573867797852 length of segment : 192 time for calcul the mask position with numpy : 0.00018978118896484375 nb_pixel_total : 2074 time to create 1 rle with old method : 0.00262451171875 length of segment : 59 time for calcul the mask position with numpy : 0.00033354759216308594 nb_pixel_total : 4552 time to create 1 rle with old method : 0.005486249923706055 length of segment : 115 time for calcul the mask position with numpy : 0.004518032073974609 nb_pixel_total : 100793 time to create 1 rle with old method : 0.1093285083770752 length of segment : 548 time for calcul the mask position with numpy : 0.004007816314697266 nb_pixel_total : 96994 time to create 1 rle with old method : 0.10360360145568848 length of segment : 464 time for calcul the mask position with numpy : 0.0007753372192382812 nb_pixel_total : 13612 time to create 1 rle with old method : 0.01772022247314453 length of segment : 199 time for calcul the mask position with numpy : 0.00019216537475585938 nb_pixel_total : 1829 time to create 1 rle with old method : 0.0022623538970947266 length of segment : 64 time for calcul the mask position with numpy : 0.0012714862823486328 nb_pixel_total : 33090 time to create 1 rle with old method : 0.040982723236083984 length of segment : 556 time for calcul the mask position with numpy : 0.0007722377777099609 nb_pixel_total : 23101 time to create 1 rle with old method : 0.02579188346862793 length of segment : 236 time for calcul the mask position with numpy : 0.0013535022735595703 nb_pixel_total : 21337 time to create 1 rle with old method : 0.034749507904052734 length of segment : 227 time for calcul the mask position with numpy : 0.0005583763122558594 nb_pixel_total : 4731 time to create 1 rle with old method : 0.006855487823486328 length of segment : 144 time for calcul the mask position with numpy : 0.0010385513305664062 nb_pixel_total : 23000 time to create 1 rle with old method : 0.027071475982666016 length of segment : 178 time for calcul the mask position with numpy : 0.0010421276092529297 nb_pixel_total : 27641 time to create 1 rle with old method : 0.03127002716064453 length of segment : 207 time for calcul the mask position with numpy : 0.003505706787109375 nb_pixel_total : 77008 time to create 1 rle with old method : 0.08744478225708008 length of segment : 488 time for calcul the mask position with numpy : 0.0017735958099365234 nb_pixel_total : 50470 time to create 1 rle with old method : 0.0551607608795166 length of segment : 288 time for calcul the mask position with numpy : 0.0007333755493164062 nb_pixel_total : 7377 time to create 1 rle with old method : 0.008665084838867188 length of segment : 218 time for calcul the mask position with numpy : 0.00027060508728027344 nb_pixel_total : 1706 time to create 1 rle with old method : 0.0021507740020751953 length of segment : 64 time for calcul the mask position with numpy : 0.00042700767517089844 nb_pixel_total : 5946 time to create 1 rle with old method : 0.008295536041259766 length of segment : 83 time for calcul the mask position with numpy : 0.00461125373840332 nb_pixel_total : 72639 time to create 1 rle with old method : 0.0845036506652832 length of segment : 435 time for calcul the mask position with numpy : 0.002706289291381836 nb_pixel_total : 33707 time to create 1 rle with old method : 0.03770613670349121 length of segment : 491 time for calcul the mask position with numpy : 0.00020074844360351562 nb_pixel_total : 1513 time to create 1 rle with old method : 0.0020127296447753906 length of segment : 47 time for calcul the mask position with numpy : 0.004823207855224609 nb_pixel_total : 141993 time to create 1 rle with old method : 0.15534758567810059 length of segment : 605 time for calcul the mask position with numpy : 0.0008230209350585938 nb_pixel_total : 18792 time to create 1 rle with old method : 0.02100086212158203 length of segment : 322 time for calcul the mask position with numpy : 0.0013039112091064453 nb_pixel_total : 58596 time to create 1 rle with old method : 0.06600666046142578 length of segment : 436 time for calcul the mask position with numpy : 0.004133939743041992 nb_pixel_total : 89006 time to create 1 rle with old method : 0.10103034973144531 length of segment : 881 time for calcul the mask position with numpy : 0.000988006591796875 nb_pixel_total : 25173 time to create 1 rle with old method : 0.03033924102783203 length of segment : 251 time for calcul the mask position with numpy : 0.000244140625 nb_pixel_total : 2244 time to create 1 rle with old method : 0.0027222633361816406 length of segment : 58 time for calcul the mask position with numpy : 0.002618551254272461 nb_pixel_total : 51572 time to create 1 rle with old method : 0.058539390563964844 length of segment : 330 time for calcul the mask position with numpy : 0.0010862350463867188 nb_pixel_total : 15248 time to create 1 rle with old method : 0.017836332321166992 length of segment : 346 time for calcul the mask position with numpy : 0.0005817413330078125 nb_pixel_total : 16654 time to create 1 rle with old method : 0.019893169403076172 length of segment : 96 time for calcul the mask position with numpy : 0.00016164779663085938 nb_pixel_total : 1925 time to create 1 rle with old method : 0.002380847930908203 length of segment : 53 time for calcul the mask position with numpy : 0.002568960189819336 nb_pixel_total : 133237 time to create 1 rle with old method : 0.14453387260437012 length of segment : 488 time for calcul the mask position with numpy : 0.0007524490356445312 nb_pixel_total : 15324 time to create 1 rle with old method : 0.026506423950195312 length of segment : 170 time for calcul the mask position with numpy : 0.0005488395690917969 nb_pixel_total : 14279 time to create 1 rle with old method : 0.01632833480834961 length of segment : 153 time for calcul the mask position with numpy : 0.00019121170043945312 nb_pixel_total : 3763 time to create 1 rle with old method : 0.004347085952758789 length of segment : 77 time for calcul the mask position with numpy : 0.00012087821960449219 nb_pixel_total : 1219 time to create 1 rle with old method : 0.0015225410461425781 length of segment : 41 time for calcul the mask position with numpy : 0.002733469009399414 nb_pixel_total : 108731 time to create 1 rle with old method : 0.12167954444885254 length of segment : 362 time for calcul the mask position with numpy : 0.0015583038330078125 nb_pixel_total : 53542 time to create 1 rle with old method : 0.05932044982910156 length of segment : 234 time for calcul the mask position with numpy : 0.0006403923034667969 nb_pixel_total : 12821 time to create 1 rle with old method : 0.017403364181518555 length of segment : 175 time for calcul the mask position with numpy : 0.0003142356872558594 nb_pixel_total : 5091 time to create 1 rle with old method : 0.006105899810791016 length of segment : 88 time for calcul the mask position with numpy : 0.002960205078125 nb_pixel_total : 101167 time to create 1 rle with old method : 0.11020255088806152 length of segment : 466 time for calcul the mask position with numpy : 0.00575709342956543 nb_pixel_total : 208538 time to create 1 rle with new method : 0.01905059814453125 length of segment : 542 time for calcul the mask position with numpy : 0.0009288787841796875 nb_pixel_total : 16829 time to create 1 rle with old method : 0.018865346908569336 length of segment : 242 time for calcul the mask position with numpy : 0.001997232437133789 nb_pixel_total : 76603 time to create 1 rle with old method : 0.0841207504272461 length of segment : 366 time for calcul the mask position with numpy : 0.0002536773681640625 nb_pixel_total : 3550 time to create 1 rle with old method : 0.0045812129974365234 length of segment : 90 time for calcul the mask position with numpy : 0.002317667007446289 nb_pixel_total : 85121 time to create 1 rle with old method : 0.09767985343933105 length of segment : 342 time for calcul the mask position with numpy : 0.003864288330078125 nb_pixel_total : 162356 time to create 1 rle with new method : 0.02426743507385254 length of segment : 462 time for calcul the mask position with numpy : 0.0021865367889404297 nb_pixel_total : 64676 time to create 1 rle with old method : 0.07154488563537598 length of segment : 412 time for calcul the mask position with numpy : 0.001689910888671875 nb_pixel_total : 18167 time to create 1 rle with old method : 0.021419286727905273 length of segment : 245 time for calcul the mask position with numpy : 0.0001842975616455078 nb_pixel_total : 1907 time to create 1 rle with old method : 0.0033164024353027344 length of segment : 87 time for calcul the mask position with numpy : 0.048268795013427734 nb_pixel_total : 116544 time to create 1 rle with old method : 0.13300395011901855 length of segment : 669 time for calcul the mask position with numpy : 0.017254114151000977 nb_pixel_total : 40416 time to create 1 rle with old method : 0.0515589714050293 length of segment : 260 time for calcul the mask position with numpy : 0.011130809783935547 nb_pixel_total : 57813 time to create 1 rle with old method : 0.06388068199157715 length of segment : 301 time for calcul the mask position with numpy : 0.004539966583251953 nb_pixel_total : 25118 time to create 1 rle with old method : 0.031186342239379883 length of segment : 177 time for calcul the mask position with numpy : 0.0006542205810546875 nb_pixel_total : 4648 time to create 1 rle with old method : 0.005542755126953125 length of segment : 121 time for calcul the mask position with numpy : 0.08852124214172363 nb_pixel_total : 113716 time to create 1 rle with old method : 0.12709259986877441 length of segment : 506 time for calcul the mask position with numpy : 0.007911920547485352 nb_pixel_total : 86499 time to create 1 rle with old method : 0.09670305252075195 length of segment : 526 time for calcul the mask position with numpy : 0.0007271766662597656 nb_pixel_total : 16662 time to create 1 rle with old method : 0.01868414878845215 length of segment : 163 time for calcul the mask position with numpy : 0.0011696815490722656 nb_pixel_total : 11450 time to create 1 rle with old method : 0.013335704803466797 length of segment : 156 time for calcul the mask position with numpy : 0.00890970230102539 nb_pixel_total : 55464 time to create 1 rle with old method : 0.06311345100402832 length of segment : 267 time for calcul the mask position with numpy : 0.004065990447998047 nb_pixel_total : 15408 time to create 1 rle with old method : 0.019755840301513672 length of segment : 133 time for calcul the mask position with numpy : 0.0004563331604003906 nb_pixel_total : 13964 time to create 1 rle with old method : 0.01671624183654785 length of segment : 97 time for calcul the mask position with numpy : 0.001379251480102539 nb_pixel_total : 54381 time to create 1 rle with old method : 0.06116223335266113 length of segment : 249 time for calcul the mask position with numpy : 0.003606081008911133 nb_pixel_total : 12888 time to create 1 rle with old method : 0.19083380699157715 length of segment : 156 time for calcul the mask position with numpy : 0.008714914321899414 nb_pixel_total : 24403 time to create 1 rle with old method : 0.030849456787109375 length of segment : 200 time for calcul the mask position with numpy : 0.027883052825927734 nb_pixel_total : 75554 time to create 1 rle with old method : 0.09395432472229004 length of segment : 361 time for calcul the mask position with numpy : 0.04700803756713867 nb_pixel_total : 202993 time to create 1 rle with new method : 0.015048027038574219 length of segment : 469 time for calcul the mask position with numpy : 0.02316117286682129 nb_pixel_total : 23133 time to create 1 rle with old method : 0.026754140853881836 length of segment : 349 time for calcul the mask position with numpy : 0.0005896091461181641 nb_pixel_total : 7002 time to create 1 rle with old method : 0.00843501091003418 length of segment : 213 time for calcul the mask position with numpy : 0.0002875328063964844 nb_pixel_total : 8358 time to create 1 rle with old method : 0.010361671447753906 length of segment : 122 time for calcul the mask position with numpy : 0.026826143264770508 nb_pixel_total : 133559 time to create 1 rle with old method : 0.15328454971313477 length of segment : 496 time for calcul the mask position with numpy : 0.0031223297119140625 nb_pixel_total : 123909 time to create 1 rle with old method : 0.1338958740234375 length of segment : 351 time for calcul the mask position with numpy : 9.918212890625e-05 nb_pixel_total : 1712 time to create 1 rle with old method : 0.0020384788513183594 length of segment : 76 time for calcul the mask position with numpy : 0.0018830299377441406 nb_pixel_total : 22768 time to create 1 rle with old method : 0.02787637710571289 length of segment : 214 time for calcul the mask position with numpy : 0.05002331733703613 nb_pixel_total : 113280 time to create 1 rle with old method : 0.12896728515625 length of segment : 554 time for calcul the mask position with numpy : 0.004375934600830078 nb_pixel_total : 99073 time to create 1 rle with old method : 0.10880374908447266 length of segment : 354 time for calcul the mask position with numpy : 0.021590232849121094 nb_pixel_total : 43750 time to create 1 rle with old method : 0.06656575202941895 length of segment : 249 time for calcul the mask position with numpy : 0.0012602806091308594 nb_pixel_total : 31713 time to create 1 rle with old method : 0.03639388084411621 length of segment : 181 time for calcul the mask position with numpy : 0.0005528926849365234 nb_pixel_total : 8905 time to create 1 rle with old method : 0.010322093963623047 length of segment : 191 time for calcul the mask position with numpy : 0.0003688335418701172 nb_pixel_total : 5112 time to create 1 rle with old method : 0.007714509963989258 length of segment : 163 time for calcul the mask position with numpy : 0.023318767547607422 nb_pixel_total : 38194 time to create 1 rle with old method : 0.045493125915527344 length of segment : 203 time for calcul the mask position with numpy : 0.005255222320556641 nb_pixel_total : 82010 time to create 1 rle with old method : 0.09294247627258301 length of segment : 317 time for calcul the mask position with numpy : 0.018258333206176758 nb_pixel_total : 53284 time to create 1 rle with old method : 0.06398510932922363 length of segment : 280 time for calcul the mask position with numpy : 0.00970768928527832 nb_pixel_total : 24720 time to create 1 rle with old method : 0.031017541885375977 length of segment : 317 time for calcul the mask position with numpy : 0.001386404037475586 nb_pixel_total : 21254 time to create 1 rle with old method : 0.024941205978393555 length of segment : 330 time for calcul the mask position with numpy : 0.05653810501098633 nb_pixel_total : 116834 time to create 1 rle with old method : 0.13474726676940918 length of segment : 409 time for calcul the mask position with numpy : 0.020833730697631836 nb_pixel_total : 15967 time to create 1 rle with old method : 0.020426511764526367 length of segment : 329 time for calcul the mask position with numpy : 0.0005934238433837891 nb_pixel_total : 7950 time to create 1 rle with old method : 0.00914311408996582 length of segment : 135 time for calcul the mask position with numpy : 0.01874375343322754 nb_pixel_total : 64982 time to create 1 rle with old method : 0.07603287696838379 length of segment : 337 time for calcul the mask position with numpy : 0.0004584789276123047 nb_pixel_total : 10193 time to create 1 rle with old method : 0.011846065521240234 length of segment : 112 time for calcul the mask position with numpy : 0.01729750633239746 nb_pixel_total : 39712 time to create 1 rle with old method : 0.04714465141296387 length of segment : 248 time for calcul the mask position with numpy : 0.0037980079650878906 nb_pixel_total : 20943 time to create 1 rle with old method : 0.02334451675415039 length of segment : 280 time for calcul the mask position with numpy : 0.006001949310302734 nb_pixel_total : 31144 time to create 1 rle with old method : 0.03748583793640137 length of segment : 276 time for calcul the mask position with numpy : 0.011673212051391602 nb_pixel_total : 70702 time to create 1 rle with old method : 0.07903480529785156 length of segment : 606 time for calcul the mask position with numpy : 0.0003638267517089844 nb_pixel_total : 3599 time to create 1 rle with old method : 0.0042994022369384766 length of segment : 135 time for calcul the mask position with numpy : 0.017786264419555664 nb_pixel_total : 36059 time to create 1 rle with old method : 0.04399228096008301 length of segment : 258 time for calcul the mask position with numpy : 0.0010614395141601562 nb_pixel_total : 2020 time to create 1 rle with old method : 0.0022835731506347656 length of segment : 78 time for calcul the mask position with numpy : 0.0008029937744140625 nb_pixel_total : 29522 time to create 1 rle with old method : 0.03275012969970703 length of segment : 189 time for calcul the mask position with numpy : 0.012899637222290039 nb_pixel_total : 26201 time to create 1 rle with old method : 0.03314828872680664 length of segment : 395 time for calcul the mask position with numpy : 0.019171714782714844 nb_pixel_total : 31767 time to create 1 rle with old method : 0.038843393325805664 length of segment : 360 time for calcul the mask position with numpy : 0.02510690689086914 nb_pixel_total : 105026 time to create 1 rle with old method : 0.11223530769348145 length of segment : 401 time for calcul the mask position with numpy : 0.0074863433837890625 nb_pixel_total : 44200 time to create 1 rle with old method : 0.0492551326751709 length of segment : 595 time for calcul the mask position with numpy : 0.014475584030151367 nb_pixel_total : 45842 time to create 1 rle with old method : 0.05426740646362305 length of segment : 340 time for calcul the mask position with numpy : 0.0015187263488769531 nb_pixel_total : 18405 time to create 1 rle with old method : 0.020146846771240234 length of segment : 170 time for calcul the mask position with numpy : 0.013419866561889648 nb_pixel_total : 23967 time to create 1 rle with old method : 0.029168128967285156 length of segment : 171 time for calcul the mask position with numpy : 0.021468639373779297 nb_pixel_total : 91038 time to create 1 rle with old method : 0.1089181900024414 length of segment : 418 time for calcul the mask position with numpy : 0.01569080352783203 nb_pixel_total : 119148 time to create 1 rle with old method : 0.13396525382995605 length of segment : 561 time for calcul the mask position with numpy : 0.04000687599182129 nb_pixel_total : 189046 time to create 1 rle with new method : 0.013278961181640625 length of segment : 680 time for calcul the mask position with numpy : 0.003862142562866211 nb_pixel_total : 6890 time to create 1 rle with old method : 0.02683734893798828 length of segment : 95 time for calcul the mask position with numpy : 0.04255318641662598 nb_pixel_total : 186932 time to create 1 rle with new method : 0.01665353775024414 length of segment : 644 time for calcul the mask position with numpy : 0.011867761611938477 nb_pixel_total : 24465 time to create 1 rle with old method : 0.030891895294189453 length of segment : 217 time for calcul the mask position with numpy : 0.012063980102539062 nb_pixel_total : 51856 time to create 1 rle with old method : 0.059827566146850586 length of segment : 281 time for calcul the mask position with numpy : 0.007973670959472656 nb_pixel_total : 30526 time to create 1 rle with old method : 0.036392927169799805 length of segment : 519 time for calcul the mask position with numpy : 0.0062007904052734375 nb_pixel_total : 151622 time to create 1 rle with new method : 0.010835409164428711 length of segment : 494 time for calcul the mask position with numpy : 0.0006089210510253906 nb_pixel_total : 12243 time to create 1 rle with old method : 0.013852119445800781 length of segment : 112 time for calcul the mask position with numpy : 0.0005087852478027344 nb_pixel_total : 4989 time to create 1 rle with old method : 0.005742788314819336 length of segment : 113 time for calcul the mask position with numpy : 0.00018405914306640625 nb_pixel_total : 1653 time to create 1 rle with old method : 0.0019800662994384766 length of segment : 50 time for calcul the mask position with numpy : 0.0013017654418945312 nb_pixel_total : 15984 time to create 1 rle with old method : 0.017428159713745117 length of segment : 232 time for calcul the mask position with numpy : 0.0007951259613037109 nb_pixel_total : 21075 time to create 1 rle with old method : 0.023077964782714844 length of segment : 187 time for calcul the mask position with numpy : 0.004338741302490234 nb_pixel_total : 72595 time to create 1 rle with old method : 0.07569098472595215 length of segment : 551 time for calcul the mask position with numpy : 0.0005133152008056641 nb_pixel_total : 3364 time to create 1 rle with old method : 0.0037627220153808594 length of segment : 94 time for calcul the mask position with numpy : 0.003119230270385742 nb_pixel_total : 38058 time to create 1 rle with old method : 0.05770301818847656 length of segment : 466 time for calcul the mask position with numpy : 0.0006651878356933594 nb_pixel_total : 14507 time to create 1 rle with old method : 0.016618013381958008 length of segment : 129 time for calcul the mask position with numpy : 0.004055500030517578 nb_pixel_total : 56708 time to create 1 rle with old method : 0.06229090690612793 length of segment : 305 time for calcul the mask position with numpy : 0.002293825149536133 nb_pixel_total : 26013 time to create 1 rle with old method : 0.03228020668029785 length of segment : 216 time for calcul the mask position with numpy : 0.011182546615600586 nb_pixel_total : 19821 time to create 1 rle with old method : 0.03239917755126953 length of segment : 187 time for calcul the mask position with numpy : 0.0013577938079833984 nb_pixel_total : 22534 time to create 1 rle with old method : 0.025368452072143555 length of segment : 225 time for calcul the mask position with numpy : 0.0008370876312255859 nb_pixel_total : 22404 time to create 1 rle with old method : 0.026288986206054688 length of segment : 188 time for calcul the mask position with numpy : 0.00039386749267578125 nb_pixel_total : 2437 time to create 1 rle with old method : 0.004597902297973633 length of segment : 84 time for calcul the mask position with numpy : 0.005068778991699219 nb_pixel_total : 63440 time to create 1 rle with old method : 0.10057592391967773 length of segment : 386 time for calcul the mask position with numpy : 0.0016672611236572266 nb_pixel_total : 42018 time to create 1 rle with old method : 0.04610466957092285 length of segment : 170 time for calcul the mask position with numpy : 0.002088308334350586 nb_pixel_total : 39582 time to create 1 rle with old method : 0.04439187049865723 length of segment : 377 time for calcul the mask position with numpy : 0.0012357234954833984 nb_pixel_total : 19578 time to create 1 rle with old method : 0.02163553237915039 length of segment : 189 time for calcul the mask position with numpy : 0.003331899642944336 nb_pixel_total : 73663 time to create 1 rle with old method : 0.08530306816101074 length of segment : 496 time for calcul the mask position with numpy : 0.00020360946655273438 nb_pixel_total : 2865 time to create 1 rle with old method : 0.0036134719848632812 length of segment : 59 time for calcul the mask position with numpy : 0.002583742141723633 nb_pixel_total : 85058 time to create 1 rle with old method : 0.0941462516784668 length of segment : 495 time for calcul the mask position with numpy : 0.0023567676544189453 nb_pixel_total : 27031 time to create 1 rle with old method : 0.02986454963684082 length of segment : 255 time for calcul the mask position with numpy : 0.001079559326171875 nb_pixel_total : 23761 time to create 1 rle with old method : 0.026993751525878906 length of segment : 237 time for calcul the mask position with numpy : 0.0008234977722167969 nb_pixel_total : 20611 time to create 1 rle with old method : 0.024542808532714844 length of segment : 230 time for calcul the mask position with numpy : 0.0003859996795654297 nb_pixel_total : 5982 time to create 1 rle with old method : 0.007077932357788086 length of segment : 91 time for calcul the mask position with numpy : 0.0007290840148925781 nb_pixel_total : 18141 time to create 1 rle with old method : 0.021144866943359375 length of segment : 227 time for calcul the mask position with numpy : 0.0013318061828613281 nb_pixel_total : 28637 time to create 1 rle with old method : 0.03291583061218262 length of segment : 274 time for calcul the mask position with numpy : 0.0016622543334960938 nb_pixel_total : 50786 time to create 1 rle with old method : 0.06066393852233887 length of segment : 408 time for calcul the mask position with numpy : 0.0012211799621582031 nb_pixel_total : 18567 time to create 1 rle with old method : 0.021494388580322266 length of segment : 163 time for calcul the mask position with numpy : 0.0015478134155273438 nb_pixel_total : 18852 time to create 1 rle with old method : 0.020561695098876953 length of segment : 280 time for calcul the mask position with numpy : 0.0003979206085205078 nb_pixel_total : 7243 time to create 1 rle with old method : 0.007845878601074219 length of segment : 100 time for calcul the mask position with numpy : 0.001161813735961914 nb_pixel_total : 30503 time to create 1 rle with old method : 0.035332679748535156 length of segment : 252 time for calcul the mask position with numpy : 0.0005230903625488281 nb_pixel_total : 6032 time to create 1 rle with old method : 0.007141590118408203 length of segment : 85 time spent for convertir_results : 15.654638051986694 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 172 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 47748 save missing photos in datou_result : time spend for datou_step_exec : 85.23539519309998 time spend to save output : 3.9386212825775146 total time spend for step 1 : 89.17401647567749 step2:crop_condition Fri Feb 21 00:16:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 172 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 14 About to insert : list_path_to_insert length 14 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 13 About to insert : list_path_to_insert length 13 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 14 About to insert : list_path_to_insert length 14 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 52 About to insert : list_path_to_insert length 52 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1338599629, 1338599434, 1338599429, 1338599177, 1338599100, 1338599097, 1338599080, 1338599077, 1338599073, 1338592819] Looping around the photos to save general results len do output : 105 /-3681203051Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203048Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203076Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203085Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203114Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203111Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203138Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203136Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203155Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203173Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203169Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557260Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557240Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557259Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203056Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203083Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203088Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203098Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203082Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203108Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203149Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203170Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557252Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557262Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557263Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557273Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557269Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203061Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203075Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203089Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203119Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203116Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203151Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203142Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203156Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203165Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203162Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203158Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203171Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203166Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203181Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203052Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203055Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203054Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203062Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203069Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203070Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203065Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203064Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203086Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203078Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203074Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203079Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203092Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203095Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203090Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203084Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203107Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203102Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203103Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203121Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203113Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203117Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203123Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203112Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203122Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203118Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203135Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203127Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203126Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203145Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203141Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203144Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203146Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203147Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203153Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203152Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203159Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203161Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203157Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203163Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203180Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203183Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203175Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203177Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203178Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203172Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557253Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557257Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557250Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557246Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557268Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203184Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203128Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203174Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557249Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557244Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203066Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203137Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203164Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557251Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203094Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3681203133Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557245Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3680557254Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599629', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599434', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599429', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599177', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599100', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599097', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599080', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599077', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599073', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338592819', None, None, None, None, None, '2601838') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 325 time used for this insertion : 0.036640167236328125 save_final save missing photos in datou_result : time spend for datou_step_exec : 50.48329758644104 time spend to save output : 0.03984642028808594 total time spend for step 2 : 50.523144006729126 step3:thcl Fri Feb 21 00:16:50 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.018573522567749023 time to convert the images to numpy array : 0.07495236396789551 time to import caffe and check if the image exist : 0.013275146484375 time to convert the images to numpy array : 0.14400196075439453 time to import caffe and check if the image exist : 0.019175052642822266 time to convert the images to numpy array : 0.13973045349121094 time to import caffe and check if the image exist : 0.01999354362487793 time to convert the images to numpy array : 0.14502811431884766 time to import caffe and check if the image exist : 0.018353939056396484 time to convert the images to numpy array : 0.14979958534240723 time to import caffe and check if the image exist : 0.016306161880493164 time to convert the images to numpy array : 0.18762946128845215 time to import caffe and check if the image exist : 0.013101339340209961 time to convert the images to numpy array : 0.19936728477478027 time to import caffe and check if the image exist : 0.015674591064453125 time to convert the images to numpy array : 0.20360684394836426 time to import caffe and check if the image exist : 0.022688865661621094 time to convert the images to numpy array : 0.19863200187683105 time to import caffe and check if the image exist : 0.017478466033935547 time to convert the images to numpy array : 0.23841166496276855 total time to convert the images to numpy array : 0.6941125392913818 list photo_ids error: [] list photo_ids correct : [-3681203164, -3680557251, -3681203094, -3681203133, -3680557245, -3680557254, -3680557257, -3680557250, -3680557246, -3680557268, -3681203184, -3681203128, -3681203174, -3680557249, -3680557244, -3681203066, -3681203137, -3680557260, -3680557240, -3680557259, -3681203056, -3681203083, -3681203088, -3681203098, -3681203082, -3681203108, -3681203149, -3681203170, -3681203051, -3681203048, -3681203076, -3681203085, -3681203114, -3681203111, -3681203138, -3681203136, -3681203155, -3681203173, -3681203169, -3680557252, -3680557262, -3680557263, -3680557273, -3680557269, -3681203061, -3681203075, -3681203089, -3681203119, -3681203116, -3681203151, -3681203159, -3681203161, -3681203157, -3681203163, -3681203180, -3681203183, -3681203175, -3681203177, -3681203178, -3681203172, -3680557253, -3681203090, -3681203084, -3681203107, -3681203102, -3681203103, -3681203121, -3681203113, -3681203117, -3681203123, -3681203112, -3681203122, -3681203062, -3681203069, -3681203070, -3681203065, -3681203064, -3681203086, -3681203078, -3681203074, -3681203079, -3681203092, -3681203095, -3681203118, -3681203135, -3681203127, -3681203126, -3681203145, -3681203141, -3681203144, -3681203146, -3681203147, -3681203153, -3681203152, -3681203142, -3681203156, -3681203165, -3681203162, -3681203158, -3681203171, -3681203166, -3681203181, -3681203052, -3681203055, -3681203054] number of photos to traite : 105 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 2131 wait 20 seconds l 3637 free memory gpu now : 2131 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1550 wait 20 seconds l 3637 free memory gpu now : 1550 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.1133899688720703 time used to do the prediction : 0.4033997058868408 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2741 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2741 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.3807590007781982 time used to do the prediction : 0.20188236236572266 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 4.315376281738281e-05 save missing photos in datou_result : time spend for datou_step_exec : 51.000789165496826 time spend to save output : 0.0015995502471923828 total time spend for step 3 : 51.00238871574402 step4:merge_mask_thcl_custom Fri Feb 21 00:17:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 172 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1338599629, 1338599434, 1338599429, 1338599177, 1338599100, 1338599097, 1338599080, 1338599077, 1338599073, 1338592819] Looping around the photos to save general results len do output : 10 /1338599629Didn't retrieve data .Didn't retrieve data . /1338599434Didn't retrieve data .Didn't retrieve data . /1338599429Didn't retrieve data .Didn't retrieve data . /1338599177Didn't retrieve data .Didn't retrieve data . /1338599100Didn't retrieve data .Didn't retrieve data . /1338599097Didn't retrieve data .Didn't retrieve data . /1338599080Didn't retrieve data .Didn't retrieve data . /1338599077Didn't retrieve data .Didn't retrieve data . /1338599073Didn't retrieve data .Didn't retrieve data . /1338592819Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599629', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599434', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599429', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599177', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599100', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599097', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599080', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599077', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599073', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338592819', None, None, None, None, None, '2601838') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.017214059829711914 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.07724499702453613 time spend to save output : 0.017920494079589844 total time spend for step 4 : 0.09516549110412598 step5:rle_unique_nms_with_priority Fri Feb 21 00:17:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 11 nb_hashtags : 5 time to prepare the origin masks : 10.308887720108032 time for calcul the mask position with numpy : 0.2876317501068115 nb_pixel_total : 6113116 time to create 1 rle with new method : 0.49340152740478516 time for calcul the mask position with numpy : 0.048732757568359375 nb_pixel_total : 335327 time to create 1 rle with new method : 0.8027210235595703 time for calcul the mask position with numpy : 0.04379701614379883 nb_pixel_total : 7425 time to create 1 rle with old method : 0.013019084930419922 time for calcul the mask position with numpy : 0.04475736618041992 nb_pixel_total : 1050 time to create 1 rle with old method : 0.0013015270233154297 time for calcul the mask position with numpy : 0.05310416221618652 nb_pixel_total : 320250 time to create 1 rle with new method : 1.294084072113037 time for calcul the mask position with numpy : 0.031275272369384766 nb_pixel_total : 36490 time to create 1 rle with old method : 0.04060482978820801 time for calcul the mask position with numpy : 0.025023460388183594 nb_pixel_total : 68461 time to create 1 rle with old method : 0.07637333869934082 time for calcul the mask position with numpy : 0.02368617057800293 nb_pixel_total : 10250 time to create 1 rle with old method : 0.015408515930175781 time for calcul the mask position with numpy : 0.02427530288696289 nb_pixel_total : 15800 time to create 1 rle with old method : 0.018276214599609375 time for calcul the mask position with numpy : 0.0219419002532959 nb_pixel_total : 2457 time to create 1 rle with old method : 0.00279998779296875 time for calcul the mask position with numpy : 0.02215099334716797 nb_pixel_total : 16090 time to create 1 rle with old method : 0.018734216690063477 time for calcul the mask position with numpy : 0.02515268325805664 nb_pixel_total : 123524 time to create 1 rle with old method : 0.1594395637512207 create new chi : 3.673064708709717 time to delete rle : 0.011905193328857422 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 8490 TO DO : save crop sub photo not yet done ! save time : 0.5516197681427002 nb_obj : 11 nb_hashtags : 6 time to prepare the origin masks : 8.5524001121521 time for calcul the mask position with numpy : 1.1695997714996338 nb_pixel_total : 6510886 time to create 1 rle with new method : 1.6921062469482422 time for calcul the mask position with numpy : 0.03462386131286621 nb_pixel_total : 109831 time to create 1 rle with old method : 0.11568427085876465 time for calcul the mask position with numpy : 0.03509998321533203 nb_pixel_total : 72083 time to create 1 rle with old method : 0.08034920692443848 time for calcul the mask position with numpy : 0.03656411170959473 nb_pixel_total : 65725 time to create 1 rle with old method : 0.0694577693939209 time for calcul the mask position with numpy : 0.03526592254638672 nb_pixel_total : 5316 time to create 1 rle with old method : 0.006081819534301758 time for calcul the mask position with numpy : 0.035794734954833984 nb_pixel_total : 51206 time to create 1 rle with old method : 0.05544424057006836 time for calcul the mask position with numpy : 0.03712177276611328 nb_pixel_total : 16914 time to create 1 rle with old method : 0.020134925842285156 time for calcul the mask position with numpy : 0.03934311866760254 nb_pixel_total : 7662 time to create 1 rle with old method : 0.008801460266113281 time for calcul the mask position with numpy : 0.027760028839111328 nb_pixel_total : 8388 time to create 1 rle with old method : 0.009458780288696289 time for calcul the mask position with numpy : 0.0222470760345459 nb_pixel_total : 5603 time to create 1 rle with old method : 0.006399869918823242 time for calcul the mask position with numpy : 0.02139425277709961 nb_pixel_total : 20745 time to create 1 rle with old method : 0.02179408073425293 time for calcul the mask position with numpy : 0.02905893325805664 nb_pixel_total : 175881 time to create 1 rle with new method : 0.9964756965637207 create new chi : 4.667627573013306 time to delete rle : 0.0004913806915283203 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 8270 TO DO : save crop sub photo not yet done ! save time : 1.3660004138946533 nb_obj : 26 nb_hashtags : 7 time to prepare the origin masks : 11.534929513931274 time for calcul the mask position with numpy : 1.5307741165161133 nb_pixel_total : 6172507 time to create 1 rle with new method : 0.9151463508605957 time for calcul the mask position with numpy : 0.03866291046142578 nb_pixel_total : 22927 time to create 1 rle with old method : 0.04120492935180664 time for calcul the mask position with numpy : 0.03920102119445801 nb_pixel_total : 13503 time to create 1 rle with old method : 0.026468276977539062 time for calcul the mask position with numpy : 0.04427742958068848 nb_pixel_total : 31651 time to create 1 rle with old method : 0.06768107414245605 time for calcul the mask position with numpy : 0.04534792900085449 nb_pixel_total : 2230 time to create 1 rle with old method : 0.00404047966003418 time for calcul the mask position with numpy : 0.04028773307800293 nb_pixel_total : 88662 time to create 1 rle with old method : 0.12287449836730957 time for calcul the mask position with numpy : 0.03286457061767578 nb_pixel_total : 1824 time to create 1 rle with old method : 0.002155780792236328 time for calcul the mask position with numpy : 0.03355097770690918 nb_pixel_total : 13414 time to create 1 rle with old method : 0.015577077865600586 time for calcul the mask position with numpy : 0.0314638614654541 nb_pixel_total : 4721 time to create 1 rle with old method : 0.005617856979370117 time for calcul the mask position with numpy : 0.04359292984008789 nb_pixel_total : 2066 time to create 1 rle with old method : 0.002899169921875 time for calcul the mask position with numpy : 0.03213977813720703 nb_pixel_total : 1687 time to create 1 rle with old method : 0.0019407272338867188 time for calcul the mask position with numpy : 0.029703378677368164 nb_pixel_total : 7324 time to create 1 rle with old method : 0.008199930191040039 time for calcul the mask position with numpy : 0.032418012619018555 nb_pixel_total : 4511 time to create 1 rle with old method : 0.005036830902099609 time for calcul the mask position with numpy : 0.03145885467529297 nb_pixel_total : 5064 time to create 1 rle with old method : 0.008616447448730469 time for calcul the mask position with numpy : 0.03693985939025879 nb_pixel_total : 72739 time to create 1 rle with old method : 0.11371326446533203 time for calcul the mask position with numpy : 0.0389561653137207 nb_pixel_total : 21284 time to create 1 rle with old method : 0.03485894203186035 time for calcul the mask position with numpy : 0.035678863525390625 nb_pixel_total : 1493 time to create 1 rle with old method : 0.001802206039428711 time for calcul the mask position with numpy : 0.030110836029052734 nb_pixel_total : 96459 time to create 1 rle with old method : 0.10749673843383789 time for calcul the mask position with numpy : 0.029247522354125977 nb_pixel_total : 5905 time to create 1 rle with old method : 0.0068721771240234375 time for calcul the mask position with numpy : 0.02963852882385254 nb_pixel_total : 31519 time to create 1 rle with old method : 0.03511166572570801 time for calcul the mask position with numpy : 0.03241682052612305 nb_pixel_total : 141772 time to create 1 rle with old method : 0.15749573707580566 time for calcul the mask position with numpy : 0.032889604568481445 nb_pixel_total : 51429 time to create 1 rle with old method : 0.06508135795593262 time for calcul the mask position with numpy : 0.035474538803100586 nb_pixel_total : 98497 time to create 1 rle with old method : 0.15374016761779785 time for calcul the mask position with numpy : 0.03772330284118652 nb_pixel_total : 274 time to create 1 rle with old method : 0.0003795623779296875 time for calcul the mask position with numpy : 0.04187512397766113 nb_pixel_total : 78797 time to create 1 rle with old method : 0.1314859390258789 time for calcul the mask position with numpy : 0.04559516906738281 nb_pixel_total : 50410 time to create 1 rle with old method : 0.0731501579284668 time for calcul the mask position with numpy : 0.03441643714904785 nb_pixel_total : 27571 time to create 1 rle with old method : 0.03075432777404785 create new chi : 4.6590259075164795 time to delete rle : 0.0009646415710449219 batch 1 Loaded 27 chid ids of type : 3726 Number RLEs to save : 15951 TO DO : save crop sub photo not yet done ! save time : 1.007049560546875 nb_obj : 10 nb_hashtags : 4 time to prepare the origin masks : 11.212124109268188 time for calcul the mask position with numpy : 1.6235601902008057 nb_pixel_total : 6686892 time to create 1 rle with new method : 1.0304498672485352 time for calcul the mask position with numpy : 0.0426945686340332 nb_pixel_total : 108820 time to create 1 rle with old method : 0.13662028312683105 time for calcul the mask position with numpy : 0.04155397415161133 nb_pixel_total : 53363 time to create 1 rle with old method : 0.05988502502441406 time for calcul the mask position with numpy : 0.04249000549316406 nb_pixel_total : 15261 time to create 1 rle with old method : 0.02625584602355957 time for calcul the mask position with numpy : 0.046004295349121094 nb_pixel_total : 133119 time to create 1 rle with old method : 0.1787419319152832 time for calcul the mask position with numpy : 0.038767099380493164 nb_pixel_total : 1210 time to create 1 rle with old method : 0.0015804767608642578 time for calcul the mask position with numpy : 0.04002976417541504 nb_pixel_total : 3754 time to create 1 rle with old method : 0.004549503326416016 time for calcul the mask position with numpy : 0.02647566795349121 nb_pixel_total : 14234 time to create 1 rle with old method : 0.017899036407470703 time for calcul the mask position with numpy : 0.025887012481689453 nb_pixel_total : 1910 time to create 1 rle with old method : 0.002302885055541992 time for calcul the mask position with numpy : 0.023908376693725586 nb_pixel_total : 16499 time to create 1 rle with old method : 0.01803445816040039 time for calcul the mask position with numpy : 0.023228168487548828 nb_pixel_total : 15178 time to create 1 rle with old method : 0.016466140747070312 create new chi : 3.5057382583618164 time to delete rle : 0.00038552284240722656 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 6176 TO DO : save crop sub photo not yet done ! save time : 0.43562889099121094 nb_obj : 15 nb_hashtags : 6 time to prepare the origin masks : 14.286497116088867 time for calcul the mask position with numpy : 0.8309333324432373 nb_pixel_total : 6099999 time to create 1 rle with new method : 0.8499093055725098 time for calcul the mask position with numpy : 0.04010772705078125 nb_pixel_total : 1530 time to create 1 rle with old method : 0.0025768280029296875 time for calcul the mask position with numpy : 0.04309582710266113 nb_pixel_total : 116326 time to create 1 rle with old method : 0.1339280605316162 time for calcul the mask position with numpy : 0.04005908966064453 nb_pixel_total : 64551 time to create 1 rle with old method : 0.0694878101348877 time for calcul the mask position with numpy : 0.03940749168395996 nb_pixel_total : 16706 time to create 1 rle with old method : 0.01822352409362793 time for calcul the mask position with numpy : 0.03692936897277832 nb_pixel_total : 101122 time to create 1 rle with old method : 0.11062765121459961 time for calcul the mask position with numpy : 0.027660846710205078 nb_pixel_total : 159469 time to create 1 rle with new method : 0.8765726089477539 time for calcul the mask position with numpy : 0.022611141204833984 nb_pixel_total : 40284 time to create 1 rle with old method : 0.0424802303314209 time for calcul the mask position with numpy : 0.021312475204467773 nb_pixel_total : 1895 time to create 1 rle with old method : 0.0022051334381103516 time for calcul the mask position with numpy : 0.023211240768432617 nb_pixel_total : 76514 time to create 1 rle with old method : 0.088043212890625 time for calcul the mask position with numpy : 0.024568796157836914 nb_pixel_total : 3514 time to create 1 rle with old method : 0.004111289978027344 time for calcul the mask position with numpy : 0.024723291397094727 nb_pixel_total : 57553 time to create 1 rle with old method : 0.06140613555908203 time for calcul the mask position with numpy : 0.02302718162536621 nb_pixel_total : 5078 time to create 1 rle with old method : 0.0057070255279541016 time for calcul the mask position with numpy : 0.02532649040222168 nb_pixel_total : 85001 time to create 1 rle with old method : 0.09636926651000977 time for calcul the mask position with numpy : 0.025539398193359375 nb_pixel_total : 207914 time to create 1 rle with new method : 0.38648223876953125 time for calcul the mask position with numpy : 0.023459434509277344 nb_pixel_total : 12784 time to create 1 rle with old method : 0.01358652114868164 create new chi : 4.1238648891448975 time to delete rle : 0.0005123615264892578 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 11494 TO DO : save crop sub photo not yet done ! save time : 0.7121129035949707 nb_obj : 8 nb_hashtags : 5 time to prepare the origin masks : 11.475995779037476 time for calcul the mask position with numpy : 0.7907862663269043 nb_pixel_total : 6714432 time to create 1 rle with new method : 0.8670697212219238 time for calcul the mask position with numpy : 0.03897881507873535 nb_pixel_total : 54172 time to create 1 rle with old method : 0.06455874443054199 time for calcul the mask position with numpy : 0.041290283203125 nb_pixel_total : 24304 time to create 1 rle with old method : 0.026396989822387695 time for calcul the mask position with numpy : 0.021700620651245117 nb_pixel_total : 11365 time to create 1 rle with old method : 0.012857437133789062 time for calcul the mask position with numpy : 0.023267507553100586 nb_pixel_total : 16534 time to create 1 rle with old method : 0.017817974090576172 time for calcul the mask position with numpy : 0.03918051719665527 nb_pixel_total : 113629 time to create 1 rle with old method : 0.14071106910705566 time for calcul the mask position with numpy : 0.042340755462646484 nb_pixel_total : 86232 time to create 1 rle with old method : 0.09603333473205566 time for calcul the mask position with numpy : 0.0357515811920166 nb_pixel_total : 4620 time to create 1 rle with old method : 0.005041599273681641 time for calcul the mask position with numpy : 0.03433513641357422 nb_pixel_total : 24952 time to create 1 rle with old method : 0.03168320655822754 create new chi : 2.374610662460327 time to delete rle : 0.00047898292541503906 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 6363 TO DO : save crop sub photo not yet done ! save time : 0.445605993270874 nb_obj : 16 nb_hashtags : 6 time to prepare the origin masks : 22.423352003097534 time for calcul the mask position with numpy : 1.013105869293213 nb_pixel_total : 6114598 time to create 1 rle with new method : 1.5616917610168457 time for calcul the mask position with numpy : 0.03872394561767578 nb_pixel_total : 38075 time to create 1 rle with old method : 0.06729316711425781 time for calcul the mask position with numpy : 0.04119467735290527 nb_pixel_total : 4416 time to create 1 rle with old method : 0.005264997482299805 time for calcul the mask position with numpy : 0.03809309005737305 nb_pixel_total : 8900 time to create 1 rle with old method : 0.009760379791259766 time for calcul the mask position with numpy : 0.03780245780944824 nb_pixel_total : 113156 time to create 1 rle with old method : 0.1365985870361328 time for calcul the mask position with numpy : 0.04201483726501465 nb_pixel_total : 31603 time to create 1 rle with old method : 0.03660154342651367 time for calcul the mask position with numpy : 0.03669142723083496 nb_pixel_total : 43562 time to create 1 rle with old method : 0.04690861701965332 time for calcul the mask position with numpy : 0.034293174743652344 nb_pixel_total : 23118 time to create 1 rle with old method : 0.027754545211791992 time for calcul the mask position with numpy : 0.03337216377258301 nb_pixel_total : 998 time to create 1 rle with old method : 0.0011646747589111328 time for calcul the mask position with numpy : 0.02166891098022461 nb_pixel_total : 133288 time to create 1 rle with old method : 0.16322565078735352 time for calcul the mask position with numpy : 0.023383140563964844 nb_pixel_total : 123541 time to create 1 rle with old method : 0.13141465187072754 time for calcul the mask position with numpy : 0.02175307273864746 nb_pixel_total : 8290 time to create 1 rle with old method : 0.009429693222045898 time for calcul the mask position with numpy : 0.020811796188354492 nb_pixel_total : 6963 time to create 1 rle with old method : 0.007510185241699219 time for calcul the mask position with numpy : 0.02257990837097168 nb_pixel_total : 75328 time to create 1 rle with old method : 0.0805971622467041 time for calcul the mask position with numpy : 0.023045778274536133 nb_pixel_total : 202849 time to create 1 rle with new method : 0.8503856658935547 time for calcul the mask position with numpy : 0.036351680755615234 nb_pixel_total : 22730 time to create 1 rle with old method : 0.026984214782714844 time for calcul the mask position with numpy : 0.0363621711730957 nb_pixel_total : 98825 time to create 1 rle with old method : 0.12700653076171875 create new chi : 4.878265142440796 time to delete rle : 0.0006747245788574219 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 11070 TO DO : save crop sub photo not yet done ! save time : 0.6714932918548584 nb_obj : 13 nb_hashtags : 5 time to prepare the origin masks : 17.73667335510254 time for calcul the mask position with numpy : 1.0441579818725586 nb_pixel_total : 6499781 time to create 1 rle with new method : 1.5080456733703613 time for calcul the mask position with numpy : 0.03670859336853027 nb_pixel_total : 31411 time to create 1 rle with old method : 0.03367805480957031 time for calcul the mask position with numpy : 0.037770986557006836 nb_pixel_total : 39506 time to create 1 rle with old method : 0.046183109283447266 time for calcul the mask position with numpy : 0.04363656044006348 nb_pixel_total : 64665 time to create 1 rle with old method : 0.07342243194580078 time for calcul the mask position with numpy : 0.03667569160461426 nb_pixel_total : 81727 time to create 1 rle with old method : 0.09509658813476562 time for calcul the mask position with numpy : 0.03704643249511719 nb_pixel_total : 53047 time to create 1 rle with old method : 0.06044626235961914 time for calcul the mask position with numpy : 0.04044795036315918 nb_pixel_total : 15943 time to create 1 rle with old method : 0.018126487731933594 time for calcul the mask position with numpy : 0.040489912033081055 nb_pixel_total : 21209 time to create 1 rle with old method : 0.0290524959564209 time for calcul the mask position with numpy : 0.04579758644104004 nb_pixel_total : 3571 time to create 1 rle with old method : 0.0052225589752197266 time for calcul the mask position with numpy : 0.042742013931274414 nb_pixel_total : 116599 time to create 1 rle with old method : 0.13874125480651855 time for calcul the mask position with numpy : 0.03479313850402832 nb_pixel_total : 19670 time to create 1 rle with old method : 0.021103382110595703 time for calcul the mask position with numpy : 0.028276681900024414 nb_pixel_total : 7925 time to create 1 rle with old method : 0.010008096694946289 time for calcul the mask position with numpy : 0.039414167404174805 nb_pixel_total : 24478 time to create 1 rle with old method : 0.028289794921875 time for calcul the mask position with numpy : 0.03656888008117676 nb_pixel_total : 70708 time to create 1 rle with old method : 0.09006381034851074 create new chi : 3.7450878620147705 time to delete rle : 0.0005655288696289062 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 10062 TO DO : save crop sub photo not yet done ! save time : 0.658402681350708 nb_obj : 17 nb_hashtags : 6 time to prepare the origin masks : 23.999516487121582 time for calcul the mask position with numpy : 2.394852638244629 nb_pixel_total : 6054840 time to create 1 rle with new method : 2.023423194885254 time for calcul the mask position with numpy : 0.04436969757080078 nb_pixel_total : 104622 time to create 1 rle with old method : 0.11587715148925781 time for calcul the mask position with numpy : 0.04380488395690918 nb_pixel_total : 26220 time to create 1 rle with old method : 0.04141974449157715 time for calcul the mask position with numpy : 0.044800519943237305 nb_pixel_total : 31492 time to create 1 rle with old method : 0.03443574905395508 time for calcul the mask position with numpy : 0.04194021224975586 nb_pixel_total : 23877 time to create 1 rle with old method : 0.028927326202392578 time for calcul the mask position with numpy : 0.0405118465423584 nb_pixel_total : 186804 time to create 1 rle with new method : 2.0189754962921143 time for calcul the mask position with numpy : 0.05340838432312012 nb_pixel_total : 24422 time to create 1 rle with old method : 0.02748703956604004 time for calcul the mask position with numpy : 0.05350160598754883 nb_pixel_total : 118746 time to create 1 rle with old method : 0.12618446350097656 time for calcul the mask position with numpy : 0.04972434043884277 nb_pixel_total : 6849 time to create 1 rle with old method : 0.007673025131225586 time for calcul the mask position with numpy : 0.04607129096984863 nb_pixel_total : 75179 time to create 1 rle with old method : 0.08074092864990234 time for calcul the mask position with numpy : 0.054691314697265625 nb_pixel_total : 188292 time to create 1 rle with new method : 0.4833242893218994 time for calcul the mask position with numpy : 0.05359935760498047 nb_pixel_total : 32333 time to create 1 rle with old method : 0.03858304023742676 time for calcul the mask position with numpy : 0.05029726028442383 nb_pixel_total : 29428 time to create 1 rle with old method : 0.03183579444885254 time for calcul the mask position with numpy : 0.05559659004211426 nb_pixel_total : 2015 time to create 1 rle with old method : 0.0022993087768554688 time for calcul the mask position with numpy : 0.0496523380279541 nb_pixel_total : 18376 time to create 1 rle with old method : 0.02001786231994629 time for calcul the mask position with numpy : 0.04844331741333008 nb_pixel_total : 45750 time to create 1 rle with old method : 0.04947781562805176 time for calcul the mask position with numpy : 0.04963850975036621 nb_pixel_total : 30413 time to create 1 rle with old method : 0.03267502784729004 time for calcul the mask position with numpy : 0.053841590881347656 nb_pixel_total : 50582 time to create 1 rle with old method : 0.05467796325683594 create new chi : 8.55782961845398 time to delete rle : 0.0010328292846679688 batch 1 Loaded 18 chid ids of type : 3726 Number RLEs to save : 13820 TO DO : save crop sub photo not yet done ! save time : 0.9139769077301025 nb_obj : 28 nb_hashtags : 9 time to prepare the origin masks : 10.01058316230774 time for calcul the mask position with numpy : 0.7656197547912598 nb_pixel_total : 6159329 time to create 1 rle with new method : 1.0876500606536865 time for calcul the mask position with numpy : 0.032297372817993164 nb_pixel_total : 12196 time to create 1 rle with old method : 0.02263355255126953 time for calcul the mask position with numpy : 0.030740022659301758 nb_pixel_total : 50665 time to create 1 rle with old method : 0.054569244384765625 time for calcul the mask position with numpy : 0.029723405838012695 nb_pixel_total : 25968 time to create 1 rle with old method : 0.027770280838012695 time for calcul the mask position with numpy : 0.030376911163330078 nb_pixel_total : 19432 time to create 1 rle with old method : 0.022183656692504883 time for calcul the mask position with numpy : 0.030205249786376953 nb_pixel_total : 26454 time to create 1 rle with old method : 0.02795577049255371 time for calcul the mask position with numpy : 0.030858755111694336 nb_pixel_total : 18066 time to create 1 rle with old method : 0.020708322525024414 time for calcul the mask position with numpy : 0.030202627182006836 nb_pixel_total : 63378 time to create 1 rle with old method : 0.0701897144317627 time for calcul the mask position with numpy : 0.029474258422851562 nb_pixel_total : 15924 time to create 1 rle with old method : 0.016878843307495117 time for calcul the mask position with numpy : 0.0302126407623291 nb_pixel_total : 56514 time to create 1 rle with old method : 0.06445050239562988 time for calcul the mask position with numpy : 0.031006336212158203 nb_pixel_total : 3359 time to create 1 rle with old method : 0.0036118030548095703 time for calcul the mask position with numpy : 0.03035259246826172 nb_pixel_total : 20979 time to create 1 rle with old method : 0.02170729637145996 time for calcul the mask position with numpy : 0.03119182586669922 nb_pixel_total : 151330 time to create 1 rle with new method : 0.7637581825256348 time for calcul the mask position with numpy : 0.03218412399291992 nb_pixel_total : 39627 time to create 1 rle with old method : 0.05840134620666504 time for calcul the mask position with numpy : 0.030259370803833008 nb_pixel_total : 2436 time to create 1 rle with old method : 0.0028600692749023438 time for calcul the mask position with numpy : 0.03085780143737793 nb_pixel_total : 22316 time to create 1 rle with old method : 0.024644851684570312 time for calcul the mask position with numpy : 0.030079364776611328 nb_pixel_total : 30449 time to create 1 rle with old method : 0.0338902473449707 time for calcul the mask position with numpy : 0.030596017837524414 nb_pixel_total : 14480 time to create 1 rle with old method : 0.01604628562927246 time for calcul the mask position with numpy : 0.030992984771728516 nb_pixel_total : 84900 time to create 1 rle with old method : 0.09265637397766113 time for calcul the mask position with numpy : 0.029846906661987305 nb_pixel_total : 220 time to create 1 rle with old method : 0.0003631114959716797 time for calcul the mask position with numpy : 0.030928373336791992 nb_pixel_total : 72354 time to create 1 rle with old method : 0.10023832321166992 time for calcul the mask position with numpy : 0.03218579292297363 nb_pixel_total : 37850 time to create 1 rle with old method : 0.04654502868652344 time for calcul the mask position with numpy : 0.03451204299926758 nb_pixel_total : 2832 time to create 1 rle with old method : 0.004717826843261719 time for calcul the mask position with numpy : 0.0335385799407959 nb_pixel_total : 4969 time to create 1 rle with old method : 0.0059587955474853516 time for calcul the mask position with numpy : 0.030452728271484375 nb_pixel_total : 1644 time to create 1 rle with old method : 0.0019025802612304688 time for calcul the mask position with numpy : 0.030417442321777344 nb_pixel_total : 19733 time to create 1 rle with old method : 0.02127981185913086 time for calcul the mask position with numpy : 0.0316004753112793 nb_pixel_total : 28588 time to create 1 rle with old method : 0.03132772445678711 time for calcul the mask position with numpy : 0.03180360794067383 nb_pixel_total : 41799 time to create 1 rle with old method : 0.04588890075683594 time for calcul the mask position with numpy : 0.03174591064453125 nb_pixel_total : 22449 time to create 1 rle with old method : 0.03197932243347168 create new chi : 4.431351900100708 time to delete rle : 0.0011746883392333984 batch 1 Loaded 30 chid ids of type : 3726 Number RLEs to save : 15651 TO DO : save crop sub photo not yet done ! save time : 2.452681541442871 map_output_result : {1338599629: (0.0, 'Should be the crop_list due to order', 0), 1338599434: (0.0, 'Should be the crop_list due to order', 0), 1338599429: (0.0, 'Should be the crop_list due to order', 0), 1338599177: (0.0, 'Should be the crop_list due to order', 0), 1338599100: (0.0, 'Should be the crop_list due to order', 0), 1338599097: (0.0, 'Should be the crop_list due to order', 0), 1338599080: (0.0, 'Should be the crop_list due to order', 0), 1338599077: (0.0, 'Should be the crop_list due to order', 0), 1338599073: (0.0, 'Should be the crop_list due to order', 0), 1338592819: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1338599629, 1338599434, 1338599429, 1338599177, 1338599100, 1338599097, 1338599080, 1338599077, 1338599073, 1338592819] Looping around the photos to save general results len do output : 10 /1338599629.Didn't retrieve data . /1338599434.Didn't retrieve data . /1338599429.Didn't retrieve data . /1338599177.Didn't retrieve data . /1338599100.Didn't retrieve data . /1338599097.Didn't retrieve data . /1338599080.Didn't retrieve data . /1338599077.Didn't retrieve data . /1338599073.Didn't retrieve data . /1338592819.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599629', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599434', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599429', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599177', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599100', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599097', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599080', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599077', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599073', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338592819', None, None, None, None, None, '2601838') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.022214651107788086 save_final save missing photos in datou_result : time spend for datou_step_exec : 196.8796193599701 time spend to save output : 0.023497819900512695 total time spend for step 5 : 196.9031171798706 step6:crop_condition Fri Feb 21 00:20:58 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 166 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740093663_4163826 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.0877151489257812 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740093668_4163826 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.8659250736236572 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740093676_4163826 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 3.2201900482177734 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 14 About to insert : list_path_to_insert length 14 new photo from crops ! About to upload 14 photos upload in portfolio : 4869462 init cache_photo without model_param we have 14 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740093684_4163826 we have uploaded 14 photos in the portfolio 4869462 time of upload the photos Elapsed time : 5.984789848327637 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740093693_4163826 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.2855446338653564 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740093699_4163826 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.9268567562103271 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1338599629, 1338599434, 1338599429, 1338599177, 1338599100, 1338599097, 1338599080, 1338599077, 1338599073, 1338592819] Looping around the photos to save general results len do output : 44 /1338695201Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695202Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695204Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695205Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695206Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695207Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695208Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695210Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695211Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695213Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695214Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695215Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695216Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695220Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695221Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695222Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695223Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695224Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695225Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695226Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695227Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695231Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695232Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695233Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695234Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695236Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695237Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695238Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695239Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695240Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695241Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695242Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695243Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695244Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695245Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695248Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695249Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695250Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695252Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695253Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695254Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695256Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695257Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338695258Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599629', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599434', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599429', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599177', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599100', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599097', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599080', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599077', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599073', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338592819', None, None, None, None, None, '2601838') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 142 time used for this insertion : 0.03524422645568848 save_final save missing photos in datou_result : time spend for datou_step_exec : 44.45124578475952 time spend to save output : 0.3211684226989746 total time spend for step 6 : 44.772414207458496 step7:ventilate_hashtags_in_portfolio Fri Feb 21 00:21:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20699412 get user id for portfolio 20699412 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20699412 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','environnement','autre_refus','plastique','Carton_brun','flou','Carton_gris','kraft','Teint_Dans_La_Masse','metal','cartonnette','mal_croppe')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20699412 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','environnement','autre_refus','plastique','Carton_brun','flou','Carton_gris','kraft','Teint_Dans_La_Masse','metal','cartonnette','mal_croppe')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20699412 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','environnement','autre_refus','plastique','Carton_brun','flou','Carton_gris','kraft','Teint_Dans_La_Masse','metal','cartonnette','mal_croppe')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20699820,20699821,20699822,20699823,20699824,20699825,20699826,20699827,20699828,20699829,20699830,20699831?tags=papier,environnement,autre_refus,plastique,Carton_brun,flou,Carton_gris,kraft,Teint_Dans_La_Masse,metal,cartonnette,mal_croppe Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1338599629, 1338599434, 1338599429, 1338599177, 1338599100, 1338599097, 1338599080, 1338599077, 1338599073, 1338592819] Looping around the photos to save general results len do output : 1 /20699412. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599629', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599434', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599429', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599177', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599100', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599097', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599080', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599077', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599073', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338592819', None, None, None, None, None, '2601838') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.017633676528930664 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.0014822483062744 time spend to save output : 0.01791071891784668 total time spend for step 7 : 1.019392967224121 step8:final Fri Feb 21 00:21:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1338599629: ('0.6814549659204545',), 1338599434: ('0.6814549659204545',), 1338599429: ('0.6814549659204545',), 1338599177: ('0.6814549659204545',), 1338599100: ('0.6814549659204545',), 1338599097: ('0.6814549659204545',), 1338599080: ('0.6814549659204545',), 1338599077: ('0.6814549659204545',), 1338599073: ('0.6814549659204545',), 1338592819: ('0.6814549659204545',)} new output for save of step final : {1338599629: ('0.6814549659204545',), 1338599434: ('0.6814549659204545',), 1338599429: ('0.6814549659204545',), 1338599177: ('0.6814549659204545',), 1338599100: ('0.6814549659204545',), 1338599097: ('0.6814549659204545',), 1338599080: ('0.6814549659204545',), 1338599077: ('0.6814549659204545',), 1338599073: ('0.6814549659204545',), 1338592819: ('0.6814549659204545',)} [1338599629, 1338599434, 1338599429, 1338599177, 1338599100, 1338599097, 1338599080, 1338599077, 1338599073, 1338592819] Looping around the photos to save general results len do output : 10 /1338599629.Didn't retrieve data . /1338599434.Didn't retrieve data . /1338599429.Didn't retrieve data . /1338599177.Didn't retrieve data . /1338599100.Didn't retrieve data . /1338599097.Didn't retrieve data . /1338599080.Didn't retrieve data . /1338599077.Didn't retrieve data . /1338599073.Didn't retrieve data . /1338592819.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599629', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599434', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599429', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599177', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599100', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599097', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599080', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599077', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599073', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338592819', None, None, None, None, None, '2601838') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.016285419464111328 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11300086975097656 time spend to save output : 0.017291784286499023 total time spend for step 8 : 0.13029265403747559 step9:velours_tree Fri Feb 21 00:21:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.15183091163635254 time spend to save output : 5.4836273193359375e-05 total time spend for step 9 : 0.1518857479095459 step10:send_mail_cod Fri Feb 21 00:21:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20699412_21-02-2025_00_21_44.pdf 20699822 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette206998221740093704 20699823 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette206998231740093705 20699825 imagette206998251740093706 20699827 change filename to text .change filename to text .change filename to text .imagette206998271740093706 20699829 imagette206998291740093706 20699830 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette206998301740093706 20699831 imagette206998311740093708 20699824 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette206998241740093708 20699826 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette206998261740093709 20699828 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette206998281740093710 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20699412 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20699820,20699821,20699822,20699823,20699824,20699825,20699826,20699827,20699828,20699829,20699830,20699831?tags=papier,environnement,autre_refus,plastique,Carton_brun,flou,Carton_gris,kraft,Teint_Dans_La_Masse,metal,cartonnette,mal_croppe your option no_mail is active, we will not send the real mail to your client args[1338599629] : ('0.6814549659204545',) no score found for photo 1338599629 We are sending mail with results at report@fotonower.com args[1338599434] : ('0.6814549659204545',) no score found for photo 1338599434 We are sending mail with results at report@fotonower.com args[1338599429] : ('0.6814549659204545',) no score found for photo 1338599429 We are sending mail with results at report@fotonower.com args[1338599177] : ('0.6814549659204545',) no score found for photo 1338599177 We are sending mail with results at report@fotonower.com args[1338599100] : ('0.6814549659204545',) no score found for photo 1338599100 We are sending mail with results at report@fotonower.com args[1338599097] : ('0.6814549659204545',) no score found for photo 1338599097 We are sending mail with results at report@fotonower.com args[1338599080] : ('0.6814549659204545',) no score found for photo 1338599080 We are sending mail with results at report@fotonower.com args[1338599077] : ('0.6814549659204545',) no score found for photo 1338599077 We are sending mail with results at report@fotonower.com args[1338599073] : ('0.6814549659204545',) no score found for photo 1338599073 We are sending mail with results at report@fotonower.com args[1338592819] : ('0.6814549659204545',) no score found for photo 1338592819 We are sending mail with results at report@fotonower.com refus_total : 0.6814549659204545 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20699412 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1338592808,1338599629,1338599434,1338599429,1338599177,1338599100,1338599097,1338599080,1338599077,1338599073,1338592819,1338592688,1338592799,1338592796,1338592775,1338592771,1338592767,1338592764,1338592760,1338592757) Found this number of photos: 20 begin to download photo : 1338592808 begin to download photo : 1338599100 begin to download photo : 1338592819 begin to download photo : 1338592771 download finish for photo 1338599100 begin to download photo : 1338599097 download finish for photo 1338592808 begin to download photo : 1338599629 download finish for photo 1338592771 begin to download photo : 1338592767 download finish for photo 1338592819 begin to download photo : 1338592688 download finish for photo 1338599097 begin to download photo : 1338599080 download finish for photo 1338592767 begin to download photo : 1338592764 download finish for photo 1338599629 begin to download photo : 1338599434 download finish for photo 1338599080 begin to download photo : 1338599077 download finish for photo 1338592688 begin to download photo : 1338592799 download finish for photo 1338592764 begin to download photo : 1338592760 download finish for photo 1338599077 begin to download photo : 1338599073 download finish for photo 1338599434 begin to download photo : 1338599429 download finish for photo 1338592799 begin to download photo : 1338592796 download finish for photo 1338592760 begin to download photo : 1338592757 download finish for photo 1338599429 begin to download photo : 1338599177 download finish for photo 1338599073 download finish for photo 1338592796 begin to download photo : 1338592775 download finish for photo 1338599177 download finish for photo 1338592757 download finish for photo 1338592775 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20699412_21-02-2025_00_21_44.pdf results_Auto_P20699412_21-02-2025_00_21_44.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20699412_21-02-2025_00_21_44.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20699412','results_Auto_P20699412_21-02-2025_00_21_44.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20699412_21-02-2025_00_21_44.pdf','pdf','','1.24','0.6814549659204545') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1338599629, 1338599434, 1338599429, 1338599177, 1338599100, 1338599097, 1338599080, 1338599077, 1338599073, 1338592819] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599629', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599434', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599429', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599177', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599100', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599097', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599080', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599077', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599073', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338592819', None, None, None, None, None, '2601838') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.017169475555419922 save_final save missing photos in datou_result : time spend for datou_step_exec : 12.618762254714966 time spend to save output : 0.017467737197875977 total time spend for step 10 : 12.636229991912842 step11:split_time_score Fri Feb 21 00:21:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('13', 21),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 20022025 20699412 Nombre de photos uploadées : 21 / 23040 (0%) 20022025 20699412 Nombre de photos taguées (types de déchets): 0 / 21 (0%) 20022025 20699412 Nombre de photos taguées (volume) : 0 / 21 (0%) elapsed_time : load_data_split_time_score 4.76837158203125e-06 elapsed_time : order_list_meta_photo_and_scores 9.059906005859375e-06 ????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0019495487213134766 elapsed_time : insert_dashboard_record_day_entry 0.028725385665893555 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699402 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699403 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699404 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699405 order by id desc limit 1 Qualite : 0.3329848280211983 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20699406_21-02-2025_00_20_50.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699406 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20699406 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699407 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699408 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699409 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699410 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699411 order by id desc limit 1 Qualite : 0.6814549659204545 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20699412_21-02-2025_00_21_44.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699412 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20699412 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699413 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699414 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699415 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699416 order by id desc limit 1 Qualite : 0.3767832473441416 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20699417_21-02-2025_00_19_03.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699417 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20699417 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699418 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699419 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699420 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699421 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699422 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699423 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699424 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699425 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699426 order by id desc limit 1 Qualite : 0.7519938324117397 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20699439_21-02-2025_00_19_28.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699439 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20699439 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699440 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699441 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699442 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699443 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699444 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699449 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699470 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699471 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699472 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699473 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20699474 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'20022025': {'nb_upload': 21, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1338599629, 1338599434, 1338599429, 1338599177, 1338599100, 1338599097, 1338599080, 1338599077, 1338599073, 1338592819] Looping around the photos to save general results len do output : 1 /20699412Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599629', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599434', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599429', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599177', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599100', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599097', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599080', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599077', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338599073', None, None, None, None, None, '2601838') ('3459', None, None, None, None, None, None, None, '2601838') ('3459', '20699412', '1338592819', None, None, None, None, None, '2601838') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01881694793701172 save_final save missing photos in datou_result : time spend for datou_step_exec : 26.895798444747925 time spend to save output : 0.019065380096435547 total time spend for step 11 : 26.91486382484436 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 165.74user 215.46system 8:01.10elapsed 79%CPU (0avgtext+0avgdata 4715152maxresident)k 715360inputs+161848outputs (22789major+13869588minor)pagefaults 0swaps