python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1529754 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 14>10 please execute split_portfolio.py -i 20277527 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2557209'] with mtr_portfolio_ids : ['20277527'] and first list_photo_ids : [] new path : /proc/1529754/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.9800982475280762 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Thu Feb 6 00:46:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3594 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-06 00:46:33.258828: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-06 00:46:33.287170: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-06 00:46:33.288856: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f709c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-06 00:46:33.288884: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-06 00:46:33.292073: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-06 00:46:33.542360: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xebc7d20 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-06 00:46:33.542413: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-06 00:46:33.543271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-06 00:46:33.543612: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-06 00:46:33.545857: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-06 00:46:33.548001: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-06 00:46:33.548322: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-06 00:46:33.550640: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-06 00:46:33.551781: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-06 00:46:33.556380: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-06 00:46:33.557424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-06 00:46:33.557497: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-06 00:46:33.558039: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-06 00:46:33.558054: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-06 00:46:33.558063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-06 00:46:33.558976: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3142 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-06 00:46:33.819077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-06 00:46:33.819208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-06 00:46:33.819226: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-06 00:46:33.819242: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-06 00:46:33.819258: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-06 00:46:33.819272: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-06 00:46:33.819287: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-06 00:46:33.819303: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-06 00:46:33.820150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-06 00:46:33.821122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-06 00:46:33.821157: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-06 00:46:33.821173: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-06 00:46:33.821187: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-06 00:46:33.821203: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-06 00:46:33.821221: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-06 00:46:33.821235: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-06 00:46:33.821249: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-06 00:46:33.822080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-06 00:46:33.822121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-06 00:46:33.822130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-06 00:46:33.822138: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-06 00:46:33.823048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3142 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-06 00:46:48.455832: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-06 00:46:48.677453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-06 00:46:50.051731: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.57G (2759720960 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-06 00:46:50.829787: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.67GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-06 00:46:50.829865: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.67GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-06 00:46:50.836742: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-06 00:46:50.836789: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 25 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 27 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 23 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 Detection mask done ! Trying to reset tf kernel 1530174 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 31 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3594 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.8317880630493164 nb_pixel_total : 4223775 time to create 1 rle with new method : 1.4057443141937256 length of segment : 4086 time for calcul the mask position with numpy : 0.0009150505065917969 nb_pixel_total : 32406 time to create 1 rle with old method : 0.03681445121765137 length of segment : 246 time for calcul the mask position with numpy : 0.0003762245178222656 nb_pixel_total : 15700 time to create 1 rle with old method : 0.017976999282836914 length of segment : 170 time for calcul the mask position with numpy : 0.002289295196533203 nb_pixel_total : 109922 time to create 1 rle with old method : 0.12403273582458496 length of segment : 663 time for calcul the mask position with numpy : 0.0022025108337402344 nb_pixel_total : 116433 time to create 1 rle with old method : 0.13245224952697754 length of segment : 378 time for calcul the mask position with numpy : 0.002872943878173828 nb_pixel_total : 193638 time to create 1 rle with new method : 0.006806135177612305 length of segment : 573 time for calcul the mask position with numpy : 0.0033965110778808594 nb_pixel_total : 132925 time to create 1 rle with old method : 0.14649701118469238 length of segment : 482 time for calcul the mask position with numpy : 0.00045037269592285156 nb_pixel_total : 12730 time to create 1 rle with old method : 0.014324665069580078 length of segment : 186 time for calcul the mask position with numpy : 0.0003323554992675781 nb_pixel_total : 9363 time to create 1 rle with old method : 0.011050939559936523 length of segment : 108 time for calcul the mask position with numpy : 0.0013380050659179688 nb_pixel_total : 49374 time to create 1 rle with old method : 0.05715465545654297 length of segment : 321 time for calcul the mask position with numpy : 0.0006716251373291016 nb_pixel_total : 22663 time to create 1 rle with old method : 0.026969194412231445 length of segment : 162 time for calcul the mask position with numpy : 0.0002751350402832031 nb_pixel_total : 4693 time to create 1 rle with old method : 0.005826473236083984 length of segment : 102 time for calcul the mask position with numpy : 0.0009965896606445312 nb_pixel_total : 37553 time to create 1 rle with old method : 0.0431973934173584 length of segment : 212 time for calcul the mask position with numpy : 0.0002639293670654297 nb_pixel_total : 4086 time to create 1 rle with old method : 0.0049245357513427734 length of segment : 103 time for calcul the mask position with numpy : 0.0003285408020019531 nb_pixel_total : 4272 time to create 1 rle with old method : 0.005232572555541992 length of segment : 119 time for calcul the mask position with numpy : 0.00015974044799804688 nb_pixel_total : 2672 time to create 1 rle with old method : 0.0034132003784179688 length of segment : 52 time for calcul the mask position with numpy : 0.0003581047058105469 nb_pixel_total : 6569 time to create 1 rle with old method : 0.007962226867675781 length of segment : 75 time for calcul the mask position with numpy : 0.00014829635620117188 nb_pixel_total : 1436 time to create 1 rle with old method : 0.00176239013671875 length of segment : 63 time for calcul the mask position with numpy : 0.001565694808959961 nb_pixel_total : 66348 time to create 1 rle with old method : 0.07400727272033691 length of segment : 297 time for calcul the mask position with numpy : 0.0004074573516845703 nb_pixel_total : 8905 time to create 1 rle with old method : 0.010171890258789062 length of segment : 186 time for calcul the mask position with numpy : 0.08662009239196777 nb_pixel_total : 2775157 time to create 1 rle with new method : 0.7901773452758789 length of segment : 3454 time for calcul the mask position with numpy : 0.003391265869140625 nb_pixel_total : 214053 time to create 1 rle with new method : 0.005473136901855469 length of segment : 511 time for calcul the mask position with numpy : 0.0001575946807861328 nb_pixel_total : 6459 time to create 1 rle with old method : 0.007884502410888672 length of segment : 79 time for calcul the mask position with numpy : 0.0005939006805419922 nb_pixel_total : 33638 time to create 1 rle with old method : 0.038060903549194336 length of segment : 242 time for calcul the mask position with numpy : 0.004204511642456055 nb_pixel_total : 344019 time to create 1 rle with new method : 0.0076389312744140625 length of segment : 556 time for calcul the mask position with numpy : 0.0005481243133544922 nb_pixel_total : 31656 time to create 1 rle with old method : 0.04994511604309082 length of segment : 244 time for calcul the mask position with numpy : 0.7048685550689697 nb_pixel_total : 3473604 time to create 1 rle with new method : 0.5778653621673584 length of segment : 5044 time for calcul the mask position with numpy : 0.0008006095886230469 nb_pixel_total : 17261 time to create 1 rle with old method : 0.020362377166748047 length of segment : 259 time for calcul the mask position with numpy : 0.018675804138183594 nb_pixel_total : 819412 time to create 1 rle with new method : 0.03680014610290527 length of segment : 2136 time for calcul the mask position with numpy : 0.00028395652770996094 nb_pixel_total : 5608 time to create 1 rle with old method : 0.00656580924987793 length of segment : 81 time for calcul the mask position with numpy : 0.0026471614837646484 nb_pixel_total : 102163 time to create 1 rle with old method : 0.11317586898803711 length of segment : 456 time for calcul the mask position with numpy : 0.0003077983856201172 nb_pixel_total : 4864 time to create 1 rle with old method : 0.008311986923217773 length of segment : 93 time for calcul the mask position with numpy : 0.0003974437713623047 nb_pixel_total : 4592 time to create 1 rle with old method : 0.008002519607543945 length of segment : 135 time for calcul the mask position with numpy : 0.000896453857421875 nb_pixel_total : 19908 time to create 1 rle with old method : 0.0241391658782959 length of segment : 219 time for calcul the mask position with numpy : 0.0024552345275878906 nb_pixel_total : 67329 time to create 1 rle with old method : 0.07546424865722656 length of segment : 466 time for calcul the mask position with numpy : 0.0037720203399658203 nb_pixel_total : 164468 time to create 1 rle with new method : 0.005848407745361328 length of segment : 472 time for calcul the mask position with numpy : 0.00014591217041015625 nb_pixel_total : 1481 time to create 1 rle with old method : 0.0019009113311767578 length of segment : 49 time for calcul the mask position with numpy : 0.0013766288757324219 nb_pixel_total : 47156 time to create 1 rle with old method : 0.053514719009399414 length of segment : 300 time for calcul the mask position with numpy : 0.0005860328674316406 nb_pixel_total : 20669 time to create 1 rle with old method : 0.023974895477294922 length of segment : 226 time for calcul the mask position with numpy : 0.0007679462432861328 nb_pixel_total : 51257 time to create 1 rle with old method : 0.058088064193725586 length of segment : 306 time for calcul the mask position with numpy : 0.0011146068572998047 nb_pixel_total : 33455 time to create 1 rle with old method : 0.03899812698364258 length of segment : 218 time for calcul the mask position with numpy : 0.002749204635620117 nb_pixel_total : 165471 time to create 1 rle with new method : 0.007080554962158203 length of segment : 470 time for calcul the mask position with numpy : 0.00018787384033203125 nb_pixel_total : 1688 time to create 1 rle with old method : 0.002183198928833008 length of segment : 76 time for calcul the mask position with numpy : 0.00115966796875 nb_pixel_total : 26020 time to create 1 rle with old method : 0.029674768447875977 length of segment : 281 time for calcul the mask position with numpy : 0.0024514198303222656 nb_pixel_total : 111078 time to create 1 rle with old method : 0.12126302719116211 length of segment : 594 time for calcul the mask position with numpy : 0.0001647472381591797 nb_pixel_total : 2187 time to create 1 rle with old method : 0.0027031898498535156 length of segment : 69 time for calcul the mask position with numpy : 0.0008473396301269531 nb_pixel_total : 16742 time to create 1 rle with old method : 0.01950812339782715 length of segment : 334 time for calcul the mask position with numpy : 0.0009455680847167969 nb_pixel_total : 25370 time to create 1 rle with old method : 0.029240131378173828 length of segment : 249 time for calcul the mask position with numpy : 0.0044574737548828125 nb_pixel_total : 147475 time to create 1 rle with old method : 0.1630260944366455 length of segment : 630 time for calcul the mask position with numpy : 0.001969575881958008 nb_pixel_total : 30077 time to create 1 rle with old method : 0.03468179702758789 length of segment : 438 time for calcul the mask position with numpy : 0.0007674694061279297 nb_pixel_total : 16677 time to create 1 rle with old method : 0.019273996353149414 length of segment : 180 time for calcul the mask position with numpy : 0.0004723072052001953 nb_pixel_total : 13228 time to create 1 rle with old method : 0.015200614929199219 length of segment : 132 time for calcul the mask position with numpy : 0.00019168853759765625 nb_pixel_total : 1887 time to create 1 rle with old method : 0.002348184585571289 length of segment : 91 time for calcul the mask position with numpy : 0.00017118453979492188 nb_pixel_total : 1833 time to create 1 rle with old method : 0.0022466182708740234 length of segment : 57 time for calcul the mask position with numpy : 0.0010368824005126953 nb_pixel_total : 22893 time to create 1 rle with old method : 0.026182889938354492 length of segment : 193 time for calcul the mask position with numpy : 0.002263307571411133 nb_pixel_total : 68671 time to create 1 rle with old method : 0.07973027229309082 length of segment : 682 time for calcul the mask position with numpy : 0.0010840892791748047 nb_pixel_total : 9798 time to create 1 rle with old method : 0.011747598648071289 length of segment : 395 time for calcul the mask position with numpy : 0.0003314018249511719 nb_pixel_total : 8245 time to create 1 rle with old method : 0.009353160858154297 length of segment : 242 time for calcul the mask position with numpy : 0.0026433467864990234 nb_pixel_total : 79753 time to create 1 rle with old method : 0.08958554267883301 length of segment : 537 time for calcul the mask position with numpy : 0.0008513927459716797 nb_pixel_total : 15707 time to create 1 rle with old method : 0.018746614456176758 length of segment : 241 time for calcul the mask position with numpy : 0.008796930313110352 nb_pixel_total : 370727 time to create 1 rle with new method : 0.011617422103881836 length of segment : 715 time for calcul the mask position with numpy : 0.001619100570678711 nb_pixel_total : 52673 time to create 1 rle with old method : 0.05914807319641113 length of segment : 305 time for calcul the mask position with numpy : 0.0013654232025146484 nb_pixel_total : 37161 time to create 1 rle with old method : 0.04282569885253906 length of segment : 192 time for calcul the mask position with numpy : 0.0005371570587158203 nb_pixel_total : 16389 time to create 1 rle with old method : 0.027957439422607422 length of segment : 320 time for calcul the mask position with numpy : 0.0008642673492431641 nb_pixel_total : 37612 time to create 1 rle with old method : 0.043354034423828125 length of segment : 186 time for calcul the mask position with numpy : 0.0009353160858154297 nb_pixel_total : 52327 time to create 1 rle with old method : 0.05839133262634277 length of segment : 307 time for calcul the mask position with numpy : 0.0046727657318115234 nb_pixel_total : 370774 time to create 1 rle with new method : 0.011116743087768555 length of segment : 733 time for calcul the mask position with numpy : 0.0012178421020507812 nb_pixel_total : 46444 time to create 1 rle with old method : 0.052866220474243164 length of segment : 349 time for calcul the mask position with numpy : 0.0033714771270751953 nb_pixel_total : 144064 time to create 1 rle with old method : 0.15784668922424316 length of segment : 552 time for calcul the mask position with numpy : 0.0008995532989501953 nb_pixel_total : 12619 time to create 1 rle with old method : 0.01470327377319336 length of segment : 221 time for calcul the mask position with numpy : 0.002392292022705078 nb_pixel_total : 48098 time to create 1 rle with old method : 0.05472970008850098 length of segment : 268 time for calcul the mask position with numpy : 0.0002765655517578125 nb_pixel_total : 4214 time to create 1 rle with old method : 0.004881143569946289 length of segment : 110 time for calcul the mask position with numpy : 0.008243799209594727 nb_pixel_total : 224917 time to create 1 rle with new method : 0.011271476745605469 length of segment : 888 time for calcul the mask position with numpy : 0.001979827880859375 nb_pixel_total : 59098 time to create 1 rle with old method : 0.0683434009552002 length of segment : 548 time for calcul the mask position with numpy : 0.0007386207580566406 nb_pixel_total : 12109 time to create 1 rle with old method : 0.014738321304321289 length of segment : 157 time for calcul the mask position with numpy : 0.002811908721923828 nb_pixel_total : 113672 time to create 1 rle with old method : 0.12604713439941406 length of segment : 417 time for calcul the mask position with numpy : 0.0009517669677734375 nb_pixel_total : 25942 time to create 1 rle with old method : 0.029265165328979492 length of segment : 273 time for calcul the mask position with numpy : 0.005877256393432617 nb_pixel_total : 211722 time to create 1 rle with new method : 0.008173942565917969 length of segment : 770 time for calcul the mask position with numpy : 0.0004069805145263672 nb_pixel_total : 9163 time to create 1 rle with old method : 0.011278390884399414 length of segment : 71 time for calcul the mask position with numpy : 0.0015254020690917969 nb_pixel_total : 48119 time to create 1 rle with old method : 0.05406808853149414 length of segment : 415 time for calcul the mask position with numpy : 0.00045800209045410156 nb_pixel_total : 11550 time to create 1 rle with old method : 0.013307809829711914 length of segment : 181 time for calcul the mask position with numpy : 0.004845380783081055 nb_pixel_total : 150389 time to create 1 rle with new method : 0.007528781890869141 length of segment : 520 time for calcul the mask position with numpy : 0.0014202594757080078 nb_pixel_total : 35335 time to create 1 rle with old method : 0.0391845703125 length of segment : 383 time for calcul the mask position with numpy : 0.000110626220703125 nb_pixel_total : 1979 time to create 1 rle with old method : 0.0026597976684570312 length of segment : 52 time for calcul the mask position with numpy : 7.104873657226562e-05 nb_pixel_total : 2099 time to create 1 rle with old method : 0.0025370121002197266 length of segment : 67 time for calcul the mask position with numpy : 0.0029757022857666016 nb_pixel_total : 94046 time to create 1 rle with old method : 0.10530447959899902 length of segment : 390 time for calcul the mask position with numpy : 0.0026633739471435547 nb_pixel_total : 142380 time to create 1 rle with old method : 0.16163253784179688 length of segment : 481 time for calcul the mask position with numpy : 0.002819061279296875 nb_pixel_total : 151431 time to create 1 rle with new method : 0.007643222808837891 length of segment : 527 time for calcul the mask position with numpy : 0.0001323223114013672 nb_pixel_total : 1493 time to create 1 rle with old method : 0.0018768310546875 length of segment : 43 time for calcul the mask position with numpy : 0.0026063919067382812 nb_pixel_total : 195997 time to create 1 rle with new method : 0.006669282913208008 length of segment : 648 time for calcul the mask position with numpy : 0.00024390220642089844 nb_pixel_total : 4622 time to create 1 rle with old method : 0.005613565444946289 length of segment : 86 time for calcul the mask position with numpy : 0.0007617473602294922 nb_pixel_total : 34054 time to create 1 rle with old method : 0.038225650787353516 length of segment : 352 time for calcul the mask position with numpy : 0.00016164779663085938 nb_pixel_total : 2357 time to create 1 rle with old method : 0.0030040740966796875 length of segment : 67 time for calcul the mask position with numpy : 0.01942276954650879 nb_pixel_total : 633994 time to create 1 rle with new method : 0.03501582145690918 length of segment : 977 time for calcul the mask position with numpy : 0.0002589225769042969 nb_pixel_total : 3601 time to create 1 rle with old method : 0.004239320755004883 length of segment : 92 time for calcul the mask position with numpy : 0.0015642642974853516 nb_pixel_total : 44175 time to create 1 rle with old method : 0.049886226654052734 length of segment : 381 time for calcul the mask position with numpy : 0.0038557052612304688 nb_pixel_total : 99925 time to create 1 rle with old method : 0.11127495765686035 length of segment : 478 time for calcul the mask position with numpy : 0.0040781497955322266 nb_pixel_total : 166530 time to create 1 rle with new method : 0.005788087844848633 length of segment : 496 time for calcul the mask position with numpy : 0.022243499755859375 nb_pixel_total : 762535 time to create 1 rle with new method : 0.05013084411621094 length of segment : 2081 time for calcul the mask position with numpy : 0.00286865234375 nb_pixel_total : 150465 time to create 1 rle with new method : 0.006036520004272461 length of segment : 529 time for calcul the mask position with numpy : 0.0011856555938720703 nb_pixel_total : 64027 time to create 1 rle with old method : 0.07076907157897949 length of segment : 306 time for calcul the mask position with numpy : 0.00426483154296875 nb_pixel_total : 153318 time to create 1 rle with new method : 0.0075910091400146484 length of segment : 560 time for calcul the mask position with numpy : 0.001844644546508789 nb_pixel_total : 49086 time to create 1 rle with old method : 0.05587601661682129 length of segment : 294 time for calcul the mask position with numpy : 0.004499197006225586 nb_pixel_total : 211341 time to create 1 rle with new method : 0.009413719177246094 length of segment : 924 time for calcul the mask position with numpy : 0.002702951431274414 nb_pixel_total : 143186 time to create 1 rle with old method : 0.15714597702026367 length of segment : 437 time for calcul the mask position with numpy : 0.00012111663818359375 nb_pixel_total : 2148 time to create 1 rle with old method : 0.0025103092193603516 length of segment : 92 time for calcul the mask position with numpy : 8.344650268554688e-05 nb_pixel_total : 1765 time to create 1 rle with old method : 0.0022842884063720703 length of segment : 60 time for calcul the mask position with numpy : 0.00658106803894043 nb_pixel_total : 282561 time to create 1 rle with new method : 0.013551473617553711 length of segment : 706 time for calcul the mask position with numpy : 0.0006279945373535156 nb_pixel_total : 9687 time to create 1 rle with old method : 0.011272192001342773 length of segment : 189 time for calcul the mask position with numpy : 0.000141143798828125 nb_pixel_total : 2262 time to create 1 rle with old method : 0.002904176712036133 length of segment : 40 time for calcul the mask position with numpy : 0.04023575782775879 nb_pixel_total : 1316124 time to create 1 rle with new method : 0.5299127101898193 length of segment : 2330 time for calcul the mask position with numpy : 0.0023102760314941406 nb_pixel_total : 104926 time to create 1 rle with old method : 0.11835694313049316 length of segment : 270 time for calcul the mask position with numpy : 0.0013437271118164062 nb_pixel_total : 46793 time to create 1 rle with old method : 0.053066253662109375 length of segment : 374 time for calcul the mask position with numpy : 0.0009748935699462891 nb_pixel_total : 60642 time to create 1 rle with old method : 0.0676259994506836 length of segment : 469 time for calcul the mask position with numpy : 0.005102396011352539 nb_pixel_total : 331297 time to create 1 rle with new method : 0.00983119010925293 length of segment : 645 time for calcul the mask position with numpy : 0.012620210647583008 nb_pixel_total : 720777 time to create 1 rle with new method : 0.3032829761505127 length of segment : 1116 time for calcul the mask position with numpy : 0.00177001953125 nb_pixel_total : 39141 time to create 1 rle with old method : 0.04452323913574219 length of segment : 511 time for calcul the mask position with numpy : 0.0012373924255371094 nb_pixel_total : 52491 time to create 1 rle with old method : 0.05972027778625488 length of segment : 382 time for calcul the mask position with numpy : 0.00017213821411132812 nb_pixel_total : 3836 time to create 1 rle with old method : 0.004500389099121094 length of segment : 122 time for calcul the mask position with numpy : 7.128715515136719e-05 nb_pixel_total : 1820 time to create 1 rle with old method : 0.0034427642822265625 length of segment : 47 time for calcul the mask position with numpy : 0.00018143653869628906 nb_pixel_total : 7267 time to create 1 rle with old method : 0.008774280548095703 length of segment : 154 time for calcul the mask position with numpy : 0.010152578353881836 nb_pixel_total : 513226 time to create 1 rle with new method : 0.019003629684448242 length of segment : 1013 time spent for convertir_results : 20.88120460510254 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 122 chid ids of type : 3663 Number RLEs to save : 57690 save missing photos in datou_result : time spend for datou_step_exec : 86.86146140098572 time spend to save output : 8.141367197036743 total time spend for step 1 : 95.00282859802246 step2:crop_condition Thu Feb 6 00:48:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 122 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 12 About to insert : list_path_to_insert length 12 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 37 About to insert : list_path_to_insert length 37 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335044905, 1335044902, 1335044899, 1335044851, 1334976478, 1334976393, 1334976390, 1334976370, 1334976153, 1334976147] Looping around the photos to save general results len do output : 71 /-3656791137Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791139Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791161Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791181Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791203Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791204Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791134Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791156Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791160Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791171Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791168Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791185Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791193Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791234Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791231Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791233Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791244Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791243Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791149Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791172Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791198Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791191Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791218Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791222Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791230Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791242Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791136Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791135Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791132Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791140Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791141Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791143Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791155Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791152Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791154Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791166Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791165Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791170Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791169Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791174Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791194Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791197Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791186Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791199Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791189Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791178Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791187Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791195Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791179Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791182Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791180Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791207Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791213Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791210Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791201Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791216Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791205Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791206Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791208Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791226Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791228Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791247Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791248Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791142Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791212Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791200Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791227Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791238Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791175Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791192Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3656791217Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044905', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044902', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044899', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044851', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976478', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976393', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976390', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976370', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976153', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976147', None, None, None, None, None, '2557209') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 223 time used for this insertion : 0.03518223762512207 save_final save missing photos in datou_result : time spend for datou_step_exec : 40.790382385253906 time spend to save output : 0.037364959716796875 total time spend for step 2 : 40.8277473449707 step3:thcl Thu Feb 6 00:48:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.011823177337646484 time to convert the images to numpy array : 2.6226043701171875e-06 time to import caffe and check if the image exist : 0.02210521697998047 time to convert the images to numpy array : 0.49564027786254883 time to import caffe and check if the image exist : 0.02215743064880371 time to convert the images to numpy array : 0.4952719211578369 time to import caffe and check if the image exist : 0.025723934173583984 time to convert the images to numpy array : 0.4946863651275635 time to import caffe and check if the image exist : 0.021739482879638672 time to convert the images to numpy array : 0.5028886795043945 time to import caffe and check if the image exist : 0.020247459411621094 time to convert the images to numpy array : 0.5153844356536865 time to import caffe and check if the image exist : 0.027910947799682617 time to convert the images to numpy array : 0.5109848976135254 time to import caffe and check if the image exist : 0.013897180557250977 time to convert the images to numpy array : 0.5274934768676758 time to import caffe and check if the image exist : 0.06862497329711914 time to convert the images to numpy array : 0.4928321838378906 time to import caffe and check if the image exist : 0.02635359764099121 time to convert the images to numpy array : 0.5359704494476318 total time to convert the images to numpy array : 0.7994875907897949 list photo_ids error: [] list photo_ids correct : [-3656791230, -3656791242, -3656791136, -3656791135, -3656791132, -3656791140, -3656791141, -3656791143, -3656791160, -3656791171, -3656791168, -3656791185, -3656791193, -3656791234, -3656791231, -3656791233, -3656791179, -3656791182, -3656791180, -3656791207, -3656791213, -3656791210, -3656791201, -3656791216, -3656791205, -3656791206, -3656791208, -3656791226, -3656791228, -3656791247, -3656791248, -3656791142, -3656791137, -3656791139, -3656791161, -3656791181, -3656791203, -3656791204, -3656791134, -3656791156, -3656791155, -3656791152, -3656791154, -3656791166, -3656791165, -3656791170, -3656791169, -3656791174, -3656791194, -3656791197, -3656791186, -3656791199, -3656791189, -3656791178, -3656791187, -3656791195, -3656791212, -3656791200, -3656791227, -3656791238, -3656791175, -3656791192, -3656791217, -3656791244, -3656791243, -3656791149, -3656791172, -3656791198, -3656791191, -3656791218, -3656791222] number of photos to traite : 71 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2795 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 2192 wait 20 seconds l 3637 free memory gpu now : 2192 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.8731074333190918 time used to do the prediction : 0.24415874481201172 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 2180 wait 20 seconds l 3637 free memory gpu now : 2180 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 2180 wait 20 seconds l 3637 free memory gpu now : 2180 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.9107942581176758 time used to do the prediction : 0.25679945945739746 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.2649765014648438e-05 save missing photos in datou_result : time spend for datou_step_exec : 70.13782477378845 time spend to save output : 0.0006170272827148438 total time spend for step 3 : 70.13844180107117 step4:merge_mask_thcl_custom Thu Feb 6 00:49:56 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 122 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1335044905, 1335044902, 1335044899, 1335044851, 1334976478, 1334976393, 1334976390, 1334976370, 1334976153, 1334976147] Looping around the photos to save general results len do output : 10 /1335044905Didn't retrieve data .Didn't retrieve data . /1335044902Didn't retrieve data .Didn't retrieve data . /1335044899Didn't retrieve data .Didn't retrieve data . /1335044851Didn't retrieve data .Didn't retrieve data . /1334976478Didn't retrieve data .Didn't retrieve data . /1334976393Didn't retrieve data .Didn't retrieve data . /1334976390Didn't retrieve data .Didn't retrieve data . /1334976370Didn't retrieve data .Didn't retrieve data . /1334976153Didn't retrieve data .Didn't retrieve data . /1334976147Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044905', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044902', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044899', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044851', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976478', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976393', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976390', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976370', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976153', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976147', None, None, None, None, None, '2557209') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.06832671165466309 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.36271071434020996 time spend to save output : 0.06886863708496094 total time spend for step 4 : 0.4315793514251709 step5:rle_unique_nms_with_priority Thu Feb 6 00:49:56 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 1 nb_hashtags : 1 time to prepare the origin masks : 2.654125213623047 time for calcul the mask position with numpy : 0.05849719047546387 nb_pixel_total : 2551531 time to create 1 rle with new method : 1.1896569728851318 time for calcul the mask position with numpy : 0.5695188045501709 nb_pixel_total : 4498709 time to create 1 rle with new method : 1.6186301708221436 create new chi : 3.4881062507629395 time to delete rle : 0.009235620498657227 batch 1 Loaded 2 chid ids of type : 3726 Number RLEs to save : 7414 TO DO : save crop sub photo not yet done ! save time : 0.8503146171569824 nb_obj : 5 nb_hashtags : 4 time to prepare the origin masks : 6.1646037101745605 time for calcul the mask position with numpy : 0.9452636241912842 nb_pixel_total : 6569240 time to create 1 rle with new method : 1.525346279144287 time for calcul the mask position with numpy : 0.035010337829589844 nb_pixel_total : 32378 time to create 1 rle with old method : 0.03583574295043945 time for calcul the mask position with numpy : 0.023321866989135742 nb_pixel_total : 123119 time to create 1 rle with old method : 0.13622140884399414 time for calcul the mask position with numpy : 0.023140907287597656 nb_pixel_total : 116238 time to create 1 rle with old method : 0.13436126708984375 time for calcul the mask position with numpy : 0.022001028060913086 nb_pixel_total : 15632 time to create 1 rle with old method : 0.01787567138671875 time for calcul the mask position with numpy : 0.025170564651489258 nb_pixel_total : 193633 time to create 1 rle with new method : 1.3905973434448242 create new chi : 4.382382154464722 time to delete rle : 0.0008466243743896484 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 5836 TO DO : save crop sub photo not yet done ! save time : 0.5251147747039795 nb_obj : 13 nb_hashtags : 5 time to prepare the origin masks : 12.641057968139648 time for calcul the mask position with numpy : 1.3491418361663818 nb_pixel_total : 6696144 time to create 1 rle with new method : 2.9252822399139404 time for calcul the mask position with numpy : 0.03494739532470703 nb_pixel_total : 37404 time to create 1 rle with old method : 0.041048288345336914 time for calcul the mask position with numpy : 0.028213024139404297 nb_pixel_total : 22626 time to create 1 rle with old method : 0.0247347354888916 time for calcul the mask position with numpy : 0.034491539001464844 nb_pixel_total : 49378 time to create 1 rle with old method : 0.053611040115356445 time for calcul the mask position with numpy : 0.034856319427490234 nb_pixel_total : 1428 time to create 1 rle with old method : 0.0017228126525878906 time for calcul the mask position with numpy : 0.03440070152282715 nb_pixel_total : 6472 time to create 1 rle with old method : 0.007570981979370117 time for calcul the mask position with numpy : 0.03525829315185547 nb_pixel_total : 2656 time to create 1 rle with old method : 0.0030527114868164062 time for calcul the mask position with numpy : 0.03471803665161133 nb_pixel_total : 4679 time to create 1 rle with old method : 0.00533294677734375 time for calcul the mask position with numpy : 0.03660154342651367 nb_pixel_total : 132929 time to create 1 rle with old method : 0.14473462104797363 time for calcul the mask position with numpy : 0.030991792678833008 nb_pixel_total : 4246 time to create 1 rle with old method : 0.004976511001586914 time for calcul the mask position with numpy : 0.021788835525512695 nb_pixel_total : 4076 time to create 1 rle with old method : 0.004656314849853516 time for calcul the mask position with numpy : 0.021336078643798828 nb_pixel_total : 9352 time to create 1 rle with old method : 0.010519027709960938 time for calcul the mask position with numpy : 0.020659208297729492 nb_pixel_total : 12689 time to create 1 rle with old method : 0.014435529708862305 time for calcul the mask position with numpy : 0.021387815475463867 nb_pixel_total : 66161 time to create 1 rle with old method : 0.07188105583190918 create new chi : 5.102224349975586 time to delete rle : 0.0007696151733398438 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 6774 TO DO : save crop sub photo not yet done ! save time : 43.25535845756531 nb_obj : 6 nb_hashtags : 3 time to prepare the origin masks : 5.4799134731292725 time for calcul the mask position with numpy : 0.13725590705871582 nb_pixel_total : 3703978 time to create 1 rle with new method : 0.67038893699646 time for calcul the mask position with numpy : 0.02411937713623047 nb_pixel_total : 213723 time to create 1 rle with new method : 0.7177555561065674 time for calcul the mask position with numpy : 0.03956198692321777 nb_pixel_total : 318379 time to create 1 rle with new method : 0.7728722095489502 time for calcul the mask position with numpy : 0.03734731674194336 nb_pixel_total : 6419 time to create 1 rle with old method : 0.007467746734619141 time for calcul the mask position with numpy : 0.08169817924499512 nb_pixel_total : 2773535 time to create 1 rle with new method : 0.9291152954101562 time for calcul the mask position with numpy : 0.03891110420227051 nb_pixel_total : 592 time to create 1 rle with old method : 0.0009534358978271484 time for calcul the mask position with numpy : 0.03674936294555664 nb_pixel_total : 33614 time to create 1 rle with old method : 0.03994560241699219 create new chi : 3.656759738922119 time to delete rle : 0.0018475055694580078 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 11763 TO DO : save crop sub photo not yet done ! save time : 2.0876927375793457 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 5.217357873916626 time for calcul the mask position with numpy : 0.09730005264282227 nb_pixel_total : 2952788 time to create 1 rle with new method : 0.9707152843475342 time for calcul the mask position with numpy : 0.1475081443786621 nb_pixel_total : 610539 time to create 1 rle with new method : 1.1176762580871582 time for calcul the mask position with numpy : 0.02990579605102539 nb_pixel_total : 16974 time to create 1 rle with old method : 0.019489049911499023 time for calcul the mask position with numpy : 0.0728917121887207 nb_pixel_total : 3469939 time to create 1 rle with new method : 1.0776951313018799 create new chi : 3.624033212661743 time to delete rle : 0.0019290447235107422 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 15557 TO DO : save crop sub photo not yet done ! save time : 2.2863552570343018 nb_obj : 16 nb_hashtags : 4 time to prepare the origin masks : 9.209400177001953 time for calcul the mask position with numpy : 0.6804673671722412 nb_pixel_total : 6520161 time to create 1 rle with new method : 0.6036624908447266 time for calcul the mask position with numpy : 0.03997039794921875 nb_pixel_total : 67380 time to create 1 rle with old method : 0.08513069152832031 time for calcul the mask position with numpy : 0.029642581939697266 nb_pixel_total : 20625 time to create 1 rle with old method : 0.024288415908813477 time for calcul the mask position with numpy : 0.03159976005554199 nb_pixel_total : 2175 time to create 1 rle with old method : 0.1461334228515625 time for calcul the mask position with numpy : 0.032213687896728516 nb_pixel_total : 232 time to create 1 rle with old method : 0.0005605220794677734 time for calcul the mask position with numpy : 0.031053543090820312 nb_pixel_total : 51239 time to create 1 rle with old method : 0.06263279914855957 time for calcul the mask position with numpy : 0.029957056045532227 nb_pixel_total : 1677 time to create 1 rle with old method : 0.0020372867584228516 time for calcul the mask position with numpy : 0.03181648254394531 nb_pixel_total : 16416 time to create 1 rle with old method : 0.01840972900390625 time for calcul the mask position with numpy : 0.03036975860595703 nb_pixel_total : 1475 time to create 1 rle with old method : 0.0018033981323242188 time for calcul the mask position with numpy : 0.037171125411987305 nb_pixel_total : 110959 time to create 1 rle with old method : 0.12405610084533691 time for calcul the mask position with numpy : 0.032505035400390625 nb_pixel_total : 19856 time to create 1 rle with old method : 0.022689104080200195 time for calcul the mask position with numpy : 0.029715299606323242 nb_pixel_total : 4557 time to create 1 rle with old method : 0.005462646484375 time for calcul the mask position with numpy : 0.030574798583984375 nb_pixel_total : 25915 time to create 1 rle with old method : 0.028837203979492188 time for calcul the mask position with numpy : 0.028595685958862305 nb_pixel_total : 4838 time to create 1 rle with old method : 0.005806684494018555 time for calcul the mask position with numpy : 0.038060903549194336 nb_pixel_total : 33415 time to create 1 rle with old method : 0.03962850570678711 time for calcul the mask position with numpy : 0.048058271408081055 nb_pixel_total : 4078 time to create 1 rle with old method : 0.008172035217285156 time for calcul the mask position with numpy : 0.0626983642578125 nb_pixel_total : 165242 time to create 1 rle with new method : 0.8878710269927979 create new chi : 3.4003283977508545 time to delete rle : 0.0018222332000732422 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 9700 TO DO : save crop sub photo not yet done ! save time : 1.8983347415924072 nb_obj : 22 nb_hashtags : 5 time to prepare the origin masks : 16.26150393486023 time for calcul the mask position with numpy : 0.5846667289733887 nb_pixel_total : 5921900 time to create 1 rle with new method : 2.06038236618042 time for calcul the mask position with numpy : 0.024869918823242188 nb_pixel_total : 1315 time to create 1 rle with old method : 0.002366781234741211 time for calcul the mask position with numpy : 0.0303952693939209 nb_pixel_total : 22830 time to create 1 rle with old method : 0.026576519012451172 time for calcul the mask position with numpy : 0.025659561157226562 nb_pixel_total : 16647 time to create 1 rle with old method : 0.01850438117980957 time for calcul the mask position with numpy : 0.0325932502746582 nb_pixel_total : 79199 time to create 1 rle with old method : 0.08916878700256348 time for calcul the mask position with numpy : 0.025594472885131836 nb_pixel_total : 74424 time to create 1 rle with old method : 0.08278870582580566 time for calcul the mask position with numpy : 0.02841043472290039 nb_pixel_total : 144068 time to create 1 rle with old method : 0.15820693969726562 time for calcul the mask position with numpy : 0.021898508071899414 nb_pixel_total : 44285 time to create 1 rle with old method : 0.05155301094055176 time for calcul the mask position with numpy : 0.02403402328491211 nb_pixel_total : 52542 time to create 1 rle with old method : 0.06161355972290039 time for calcul the mask position with numpy : 0.02159905433654785 nb_pixel_total : 15625 time to create 1 rle with old method : 0.017733335494995117 time for calcul the mask position with numpy : 0.03228640556335449 nb_pixel_total : 8208 time to create 1 rle with old method : 0.013282299041748047 time for calcul the mask position with numpy : 0.02363872528076172 nb_pixel_total : 36836 time to create 1 rle with old method : 0.040978431701660156 time for calcul the mask position with numpy : 0.02277827262878418 nb_pixel_total : 147021 time to create 1 rle with old method : 0.1726546287536621 time for calcul the mask position with numpy : 0.022576570510864258 nb_pixel_total : 1819 time to create 1 rle with old method : 0.002157926559448242 time for calcul the mask position with numpy : 0.02495098114013672 nb_pixel_total : 4713 time to create 1 rle with old method : 0.006184816360473633 time for calcul the mask position with numpy : 0.0214383602142334 nb_pixel_total : 13205 time to create 1 rle with old method : 0.014934062957763672 time for calcul the mask position with numpy : 0.021722793579101562 nb_pixel_total : 1879 time to create 1 rle with old method : 0.0022630691528320312 time for calcul the mask position with numpy : 0.025343656539916992 nb_pixel_total : 370457 time to create 1 rle with new method : 0.6320667266845703 time for calcul the mask position with numpy : 0.03440451622009277 nb_pixel_total : 20453 time to create 1 rle with old method : 0.022388935089111328 time for calcul the mask position with numpy : 0.035764217376708984 nb_pixel_total : 16417 time to create 1 rle with old method : 0.018053770065307617 time for calcul the mask position with numpy : 0.03584885597229004 nb_pixel_total : 16685 time to create 1 rle with old method : 0.019199132919311523 time for calcul the mask position with numpy : 0.04545783996582031 nb_pixel_total : 9718 time to create 1 rle with old method : 0.013647317886352539 time for calcul the mask position with numpy : 0.04448962211608887 nb_pixel_total : 29994 time to create 1 rle with old method : 0.03351259231567383 create new chi : 4.854445457458496 time to delete rle : 0.0017960071563720703 batch 1 Loaded 23 chid ids of type : 3726 Number RLEs to save : 15998 TO DO : save crop sub photo not yet done ! save time : 2.148557662963867 nb_obj : 23 nb_hashtags : 6 time to prepare the origin masks : 21.36933732032776 time for calcul the mask position with numpy : 0.4492781162261963 nb_pixel_total : 5964294 time to create 1 rle with new method : 1.0828094482421875 time for calcul the mask position with numpy : 0.06889796257019043 nb_pixel_total : 12033 time to create 1 rle with old method : 0.01330256462097168 time for calcul the mask position with numpy : 0.05509686470031738 nb_pixel_total : 113556 time to create 1 rle with old method : 0.14838552474975586 time for calcul the mask position with numpy : 0.0517733097076416 nb_pixel_total : 25919 time to create 1 rle with old method : 0.029371976852416992 time for calcul the mask position with numpy : 0.12480616569519043 nb_pixel_total : 47928 time to create 1 rle with old method : 0.05377793312072754 time for calcul the mask position with numpy : 0.042265892028808594 nb_pixel_total : 22606 time to create 1 rle with old method : 0.025058507919311523 time for calcul the mask position with numpy : 0.048148155212402344 nb_pixel_total : 12615 time to create 1 rle with old method : 0.013947010040283203 time for calcul the mask position with numpy : 0.0433804988861084 nb_pixel_total : 4587 time to create 1 rle with old method : 0.005158901214599609 time for calcul the mask position with numpy : 0.04298138618469238 nb_pixel_total : 195507 time to create 1 rle with new method : 0.6824657917022705 time for calcul the mask position with numpy : 0.07143688201904297 nb_pixel_total : 62359 time to create 1 rle with old method : 0.06675910949707031 time for calcul the mask position with numpy : 0.03835725784301758 nb_pixel_total : 1482 time to create 1 rle with old method : 0.0023267269134521484 time for calcul the mask position with numpy : 0.030151844024658203 nb_pixel_total : 2520 time to create 1 rle with old method : 0.00301361083984375 time for calcul the mask position with numpy : 0.026280879974365234 nb_pixel_total : 2094 time to create 1 rle with old method : 0.0023794174194335938 time for calcul the mask position with numpy : 0.02804875373840332 nb_pixel_total : 93889 time to create 1 rle with old method : 0.10765647888183594 time for calcul the mask position with numpy : 0.025201082229614258 nb_pixel_total : 1954 time to create 1 rle with old method : 0.002324342727661133 time for calcul the mask position with numpy : 0.02359294891357422 nb_pixel_total : 11511 time to create 1 rle with old method : 0.01307988166809082 time for calcul the mask position with numpy : 0.02943587303161621 nb_pixel_total : 9078 time to create 1 rle with old method : 0.01013946533203125 time for calcul the mask position with numpy : 0.025278091430664062 nb_pixel_total : 48062 time to create 1 rle with old method : 0.06489706039428711 time for calcul the mask position with numpy : 0.026235580444335938 nb_pixel_total : 33861 time to create 1 rle with old method : 0.0451357364654541 time for calcul the mask position with numpy : 0.024100065231323242 nb_pixel_total : 4206 time to create 1 rle with old method : 0.005090951919555664 time for calcul the mask position with numpy : 0.02623438835144043 nb_pixel_total : 2292 time to create 1 rle with old method : 0.007863044738769531 time for calcul the mask position with numpy : 0.024694442749023438 nb_pixel_total : 11393 time to create 1 rle with old method : 0.013779640197753906 time for calcul the mask position with numpy : 0.02487039566040039 nb_pixel_total : 142174 time to create 1 rle with old method : 0.15064430236816406 time for calcul the mask position with numpy : 0.025602102279663086 nb_pixel_total : 224320 time to create 1 rle with new method : 0.630711555480957 create new chi : 4.694310426712036 time to delete rle : 0.009571075439453125 batch 1 Loaded 24 chid ids of type : 3726 Number RLEs to save : 15498 TO DO : save crop sub photo not yet done ! save time : 1.8460519313812256 nb_obj : 15 nb_hashtags : 4 time to prepare the origin masks : 18.889245748519897 time for calcul the mask position with numpy : 0.29968810081481934 nb_pixel_total : 4649707 time to create 1 rle with new method : 0.39705944061279297 time for calcul the mask position with numpy : 0.038453102111816406 nb_pixel_total : 1753 time to create 1 rle with old method : 0.002178668975830078 time for calcul the mask position with numpy : 0.043965816497802734 nb_pixel_total : 2139 time to create 1 rle with old method : 0.002613544464111328 time for calcul the mask position with numpy : 0.044663429260253906 nb_pixel_total : 46719 time to create 1 rle with old method : 0.07517004013061523 time for calcul the mask position with numpy : 0.05841350555419922 nb_pixel_total : 144370 time to create 1 rle with old method : 0.16633343696594238 time for calcul the mask position with numpy : 0.04639005661010742 nb_pixel_total : 152945 time to create 1 rle with new method : 0.37800168991088867 time for calcul the mask position with numpy : 0.04328584671020508 nb_pixel_total : 755554 time to create 1 rle with new method : 0.3624577522277832 time for calcul the mask position with numpy : 0.0434870719909668 nb_pixel_total : 282121 time to create 1 rle with new method : 0.40630531311035156 time for calcul the mask position with numpy : 0.03857779502868652 nb_pixel_total : 3586 time to create 1 rle with old method : 0.004492521286010742 time for calcul the mask position with numpy : 0.04696345329284668 nb_pixel_total : 633563 time to create 1 rle with new method : 0.5090889930725098 time for calcul the mask position with numpy : 0.025300025939941406 nb_pixel_total : 99617 time to create 1 rle with old method : 0.11013388633728027 time for calcul the mask position with numpy : 0.02381110191345215 nb_pixel_total : 2353 time to create 1 rle with old method : 0.0028007030487060547 time for calcul the mask position with numpy : 0.026970624923706055 nb_pixel_total : 17009 time to create 1 rle with old method : 0.0193939208984375 time for calcul the mask position with numpy : 0.03247785568237305 nb_pixel_total : 43992 time to create 1 rle with old method : 0.049161434173583984 time for calcul the mask position with numpy : 0.03754544258117676 nb_pixel_total : 150856 time to create 1 rle with new method : 0.44399189949035645 time for calcul the mask position with numpy : 0.03867340087890625 nb_pixel_total : 63956 time to create 1 rle with old method : 0.07395195960998535 create new chi : 4.0564000606536865 time to delete rle : 0.0017619132995605469 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 17008 TO DO : save crop sub photo not yet done ! save time : 2.7644567489624023 nb_obj : 14 nb_hashtags : 4 time to prepare the origin masks : 12.05453634262085 time for calcul the mask position with numpy : 1.2765517234802246 nb_pixel_total : 4365168 time to create 1 rle with new method : 1.377718448638916 time for calcul the mask position with numpy : 0.04975175857543945 nb_pixel_total : 46666 time to create 1 rle with old method : 0.0857076644897461 time for calcul the mask position with numpy : 0.04854702949523926 nb_pixel_total : 382405 time to create 1 rle with new method : 0.6544878482818604 time for calcul the mask position with numpy : 0.0361173152923584 nb_pixel_total : 7227 time to create 1 rle with old method : 0.008212089538574219 time for calcul the mask position with numpy : 0.0689699649810791 nb_pixel_total : 1819 time to create 1 rle with old method : 0.006978511810302734 time for calcul the mask position with numpy : 0.03827834129333496 nb_pixel_total : 3809 time to create 1 rle with old method : 0.004602670669555664 time for calcul the mask position with numpy : 0.04155921936035156 nb_pixel_total : 104466 time to create 1 rle with old method : 0.11945223808288574 time for calcul the mask position with numpy : 0.04420661926269531 nb_pixel_total : 442558 time to create 1 rle with new method : 0.4302678108215332 time for calcul the mask position with numpy : 0.04106736183166504 nb_pixel_total : 238429 time to create 1 rle with new method : 0.7040023803710938 time for calcul the mask position with numpy : 0.06092572212219238 nb_pixel_total : 1304147 time to create 1 rle with new method : 0.5294580459594727 time for calcul the mask position with numpy : 0.04410505294799805 nb_pixel_total : 60642 time to create 1 rle with old method : 0.06979846954345703 time for calcul the mask position with numpy : 0.04198884963989258 nb_pixel_total : 2234 time to create 1 rle with old method : 0.0025916099548339844 time for calcul the mask position with numpy : 0.03792548179626465 nb_pixel_total : 9633 time to create 1 rle with old method : 0.011863231658935547 time for calcul the mask position with numpy : 0.03761124610900879 nb_pixel_total : 28665 time to create 1 rle with old method : 0.03333449363708496 time for calcul the mask position with numpy : 0.03834962844848633 nb_pixel_total : 52372 time to create 1 rle with old method : 0.06457281112670898 create new chi : 6.1652703285217285 time to delete rle : 0.0022840499877929688 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 15740 TO DO : save crop sub photo not yet done ! save time : 3.564242124557495 map_output_result : {1335044905: (0.0, 'Should be the crop_list due to order', 0.0), 1335044902: (0.0, 'Should be the crop_list due to order', 0.0), 1335044899: (0.0, 'Should be the crop_list due to order', 0.0), 1335044851: (0.0, 'Should be the crop_list due to order', 0.0), 1334976478: (0.0, 'Should be the crop_list due to order', 0.0), 1334976393: (0.0, 'Should be the crop_list due to order', 0.0), 1334976390: (0.0, 'Should be the crop_list due to order', 0.0), 1334976370: (0.0, 'Should be the crop_list due to order', 0.0), 1334976153: (0.0, 'Should be the crop_list due to order', 0.0), 1334976147: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1335044905, 1335044902, 1335044899, 1335044851, 1334976478, 1334976393, 1334976390, 1334976370, 1334976153, 1334976147] Looping around the photos to save general results len do output : 10 /1335044905.Didn't retrieve data . /1335044902.Didn't retrieve data . /1335044899.Didn't retrieve data . /1335044851.Didn't retrieve data . /1334976478.Didn't retrieve data . /1334976393.Didn't retrieve data . /1334976390.Didn't retrieve data . /1334976370.Didn't retrieve data . /1334976153.Didn't retrieve data . /1334976147.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044905', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044902', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044899', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044851', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976478', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976393', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976390', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976370', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976153', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976147', None, None, None, None, None, '2557209') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.020311355590820312 save_final save missing photos in datou_result : time spend for datou_step_exec : 215.78437614440918 time spend to save output : 0.19742989540100098 total time spend for step 5 : 215.98180603981018 step6:crop_condition Thu Feb 6 00:53:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 128 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738799615_1529754 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.987567663192749 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738799619_1529754 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9401459693908691 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738799622_1529754 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9389588832855225 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738799626_1529754 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.8474969863891602 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738799631_1529754 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.4230821132659912 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335044905, 1335044902, 1335044899, 1335044851, 1334976478, 1334976393, 1334976390, 1334976370, 1334976153, 1334976147] Looping around the photos to save general results len do output : 14 /1335060711Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335060730Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335060731Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335060733Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335060734Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335060737Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335060738Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335060739Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335060740Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335060741Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335060742Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335060744Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335060745Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335060746Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044905', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044902', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044899', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044851', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976478', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976393', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976390', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976370', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976153', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976147', None, None, None, None, None, '2557209') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 52 time used for this insertion : 0.2228398323059082 save_final save missing photos in datou_result : time spend for datou_step_exec : 22.071494102478027 time spend to save output : 0.2234814167022705 total time spend for step 6 : 22.294975519180298 step7:ventilate_hashtags_in_portfolio Thu Feb 6 00:53:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20277527 get user id for portfolio 20277527 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20277527 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','mal_croppe','environnement','cartonnette','Teint_Dans_La_Masse','metal','kraft','plastique','papier','Carton_gris','autre_refus','Carton_brun')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20277527 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','mal_croppe','environnement','cartonnette','Teint_Dans_La_Masse','metal','kraft','plastique','papier','Carton_gris','autre_refus','Carton_brun')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20277527 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','mal_croppe','environnement','cartonnette','Teint_Dans_La_Masse','metal','kraft','plastique','papier','Carton_gris','autre_refus','Carton_brun')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20277593,20277594,20277595,20277596,20277597,20277598,20277599,20277600,20277601,20277602,20277603,20277604?tags=flou,mal_croppe,environnement,cartonnette,Teint_Dans_La_Masse,metal,kraft,plastique,papier,Carton_gris,autre_refus,Carton_brun Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1335044905, 1335044902, 1335044899, 1335044851, 1334976478, 1334976393, 1334976390, 1334976370, 1334976153, 1334976147] Looping around the photos to save general results len do output : 1 /20277527. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044905', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044902', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044899', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044851', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976478', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976393', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976390', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976370', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976153', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976147', None, None, None, None, None, '2557209') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.13375258445739746 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.7777342796325684 time spend to save output : 0.13407492637634277 total time spend for step 7 : 1.9118092060089111 step8:final Thu Feb 6 00:53:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1335044905: ('0.06769655641860853',), 1335044902: ('0.06769655641860853',), 1335044899: ('0.06769655641860853',), 1335044851: ('0.06769655641860853',), 1334976478: ('0.06769655641860853',), 1334976393: ('0.06769655641860853',), 1334976390: ('0.06769655641860853',), 1334976370: ('0.06769655641860853',), 1334976153: ('0.06769655641860853',), 1334976147: ('0.06769655641860853',)} new output for save of step final : {1335044905: ('0.06769655641860853',), 1335044902: ('0.06769655641860853',), 1335044899: ('0.06769655641860853',), 1335044851: ('0.06769655641860853',), 1334976478: ('0.06769655641860853',), 1334976393: ('0.06769655641860853',), 1334976390: ('0.06769655641860853',), 1334976370: ('0.06769655641860853',), 1334976153: ('0.06769655641860853',), 1334976147: ('0.06769655641860853',)} [1335044905, 1335044902, 1335044899, 1335044851, 1334976478, 1334976393, 1334976390, 1334976370, 1334976153, 1334976147] Looping around the photos to save general results len do output : 10 /1335044905.Didn't retrieve data . /1335044902.Didn't retrieve data . /1335044899.Didn't retrieve data . /1335044851.Didn't retrieve data . /1334976478.Didn't retrieve data . /1334976393.Didn't retrieve data . /1334976390.Didn't retrieve data . /1334976370.Didn't retrieve data . /1334976153.Didn't retrieve data . /1334976147.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044905', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044902', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044899', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044851', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976478', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976393', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976390', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976370', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976153', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976147', None, None, None, None, None, '2557209') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.04985499382019043 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.20397520065307617 time spend to save output : 0.050720930099487305 total time spend for step 8 : 0.2546961307525635 step9:velours_tree Thu Feb 6 00:53:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.18889307975769043 time spend to save output : 6.222724914550781e-05 total time spend for step 9 : 0.18895530700683594 step10:send_mail_cod Thu Feb 6 00:53:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20277527_06-02-2025_00_53_57.pdf 20277593 imagette202775931738799637 20277594 imagette202775941738799637 20277596 change filename to text .change filename to text .imagette202775961738799637 20277598 imagette202775981738799637 20277599 change filename to text .change filename to text .change filename to text .imagette202775991738799637 20277600 imagette202776001738799637 20277603 change filename to text .imagette202776031738799637 20277597 imagette202775971738799637 20277602 change filename to text .change filename to text .imagette202776021738799637 20277604 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202776041738799638 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20277527 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20277593,20277594,20277595,20277596,20277597,20277598,20277599,20277600,20277601,20277602,20277603,20277604?tags=flou,mal_croppe,environnement,cartonnette,Teint_Dans_La_Masse,metal,kraft,plastique,papier,Carton_gris,autre_refus,Carton_brun your option no_mail is active, we will not send the real mail to your client args[1335044905] : ('0.06769655641860853',) no score found for photo 1335044905 apple ('0.06769655641860853',) We are sending mail with results at report@fotonower.com args[1335044902] : ('0.06769655641860853',) no score found for photo 1335044902 apple ('0.06769655641860853',) We are sending mail with results at report@fotonower.com args[1335044899] : ('0.06769655641860853',) no score found for photo 1335044899 apple ('0.06769655641860853',) We are sending mail with results at report@fotonower.com args[1335044851] : ('0.06769655641860853',) no score found for photo 1335044851 apple ('0.06769655641860853',) We are sending mail with results at report@fotonower.com args[1334976478] : ('0.06769655641860853',) no score found for photo 1334976478 apple ('0.06769655641860853',) We are sending mail with results at report@fotonower.com args[1334976393] : ('0.06769655641860853',) no score found for photo 1334976393 apple ('0.06769655641860853',) We are sending mail with results at report@fotonower.com args[1334976390] : ('0.06769655641860853',) no score found for photo 1334976390 apple ('0.06769655641860853',) We are sending mail with results at report@fotonower.com args[1334976370] : ('0.06769655641860853',) no score found for photo 1334976370 apple ('0.06769655641860853',) We are sending mail with results at report@fotonower.com args[1334976153] : ('0.06769655641860853',) no score found for photo 1334976153 apple ('0.06769655641860853',) We are sending mail with results at report@fotonower.com args[1334976147] : ('0.06769655641860853',) no score found for photo 1334976147 apple ('0.06769655641860853',) We are sending mail with results at report@fotonower.com refus_total : 0.06769655641860853 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20277527 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1334976141,1334976143,1334976144,1334976145,1334976153,1334976390,1334976393,1335044899,1335044902,1334976147,1334976370,1334976478,1335044851,1335044905) Found this number of photos: 14 begin to download photo : 1334976141 begin to download photo : 1334976145 begin to download photo : 1334976393 begin to download photo : 1334976147 begin to download photo : 1335044851 download finish for photo 1335044851 begin to download photo : 1335044905 download finish for photo 1334976147 begin to download photo : 1334976370 download finish for photo 1334976145 begin to download photo : 1334976153 download finish for photo 1334976141 begin to download photo : 1334976143 download finish for photo 1334976393 begin to download photo : 1335044899 download finish for photo 1335044905 download finish for photo 1334976143 begin to download photo : 1334976144 download finish for photo 1334976153 begin to download photo : 1334976390 download finish for photo 1334976370 begin to download photo : 1334976478 download finish for photo 1335044899 begin to download photo : 1335044902 download finish for photo 1334976144 download finish for photo 1334976478 download finish for photo 1335044902 download finish for photo 1334976390 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20277527_06-02-2025_00_53_57.pdf results_Auto_P20277527_06-02-2025_00_53_57.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20277527_06-02-2025_00_53_57.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20277527','results_Auto_P20277527_06-02-2025_00_53_57.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20277527_06-02-2025_00_53_57.pdf','pdf','','0.55','0.06769655641860853') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1335044905, 1335044902, 1335044899, 1335044851, 1334976478, 1334976393, 1334976390, 1334976370, 1334976153, 1334976147] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044905', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044902', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044899', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044851', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976478', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976393', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976390', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976370', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976153', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976147', None, None, None, None, None, '2557209') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.13360238075256348 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.7787656784057617 time spend to save output : 0.13387250900268555 total time spend for step 10 : 3.9126381874084473 step11:split_time_score Thu Feb 6 00:54:01 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('16', 14),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 05022025 20277527 Nombre de photos uploadées : 14 / 23040 (0%) 05022025 20277527 Nombre de photos taguées (types de déchets): 0 / 14 (0%) 05022025 20277527 Nombre de photos taguées (volume) : 0 / 14 (0%) elapsed_time : load_data_split_time_score 1.9073486328125e-06 elapsed_time : order_list_meta_photo_and_scores 6.198883056640625e-06 ?????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0009467601776123047 elapsed_time : insert_dashboard_record_day_entry 0.02325606346130371 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277499 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277500 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277501 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277502 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277503 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277504 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277505 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277506 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277507 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277508 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277509 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277510 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277511 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277512 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277513 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277514 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277515 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277516 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277517 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277518 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277519 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277520 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277521 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277522 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277523 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277524 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277525 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277526 order by id desc limit 1 Qualite : 0.06769655641860853 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20277527_06-02-2025_00_53_57.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277527 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20277527 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277528 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277529 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277530 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277531 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277532 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277533 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277534 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277535 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277548 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277549 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277550 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277551 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20277552 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'05022025': {'nb_upload': 14, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1335044905, 1335044902, 1335044899, 1335044851, 1334976478, 1334976393, 1334976390, 1334976370, 1334976153, 1334976147] Looping around the photos to save general results len do output : 1 /20277527Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044905', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044902', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044899', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1335044851', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976478', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976393', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976390', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976370', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976153', None, None, None, None, None, '2557209') ('3459', None, None, None, None, None, None, None, '2557209') ('3459', '20277527', '1334976147', None, None, None, None, None, '2557209') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.015205621719360352 save_final save missing photos in datou_result : time spend for datou_step_exec : 31.839905261993408 time spend to save output : 0.015584945678710938 total time spend for step 11 : 31.85549020767212 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 140.33user 181.86system 8:09.85elapsed 65%CPU (0avgtext+0avgdata 4213924maxresident)k 573152inputs+109104outputs (20178major+11954932minor)pagefaults 0swaps