python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 923107 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 30>10 please execute split_portfolio.py -i 20314846 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2560904'] with mtr_portfolio_ids : ['20314846'] and first list_photo_ids : [] new path : /proc/923107/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.9235329627990723 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 7 01:32:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10772 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-07 01:32:35.359201: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-07 01:32:35.399159: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-07 01:32:35.401733: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f9ed8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-07 01:32:35.401789: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-07 01:32:35.407868: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-07 01:32:35.667931: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x228b6140 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-07 01:32:35.667988: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-07 01:32:35.669857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 01:32:35.670611: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 01:32:35.676197: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 01:32:35.699025: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 01:32:35.700704: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 01:32:35.743598: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 01:32:35.749601: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 01:32:35.815916: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 01:32:35.818312: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 01:32:35.818880: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 01:32:35.820473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-07 01:32:35.820502: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-07 01:32:35.820518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-07 01:32:35.822597: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9983 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-07 01:32:36.157876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 01:32:36.158022: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 01:32:36.158045: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 01:32:36.158064: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 01:32:36.158082: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 01:32:36.158099: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 01:32:36.158117: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 01:32:36.158135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 01:32:36.159691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 01:32:36.160875: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 01:32:36.160909: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 01:32:36.160924: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 01:32:36.160938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 01:32:36.160952: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 01:32:36.160971: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 01:32:36.160985: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 01:32:36.160999: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 01:32:36.162212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 01:32:36.162246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-07 01:32:36.162254: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-07 01:32:36.162261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-07 01:32:36.163578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9983 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-07 01:32:45.047574: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 01:32:45.386470: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 32 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 26 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 25 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 21 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 31 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 27 Detection mask done ! Trying to reset tf kernel 923532 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5483 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10772 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0032498836517333984 nb_pixel_total : 117338 time to create 1 rle with old method : 0.12529921531677246 length of segment : 503 time for calcul the mask position with numpy : 0.0014903545379638672 nb_pixel_total : 96841 time to create 1 rle with old method : 0.1073143482208252 length of segment : 308 time for calcul the mask position with numpy : 0.0006070137023925781 nb_pixel_total : 41355 time to create 1 rle with old method : 0.045747995376586914 length of segment : 319 time for calcul the mask position with numpy : 0.00011348724365234375 nb_pixel_total : 3441 time to create 1 rle with old method : 0.004286289215087891 length of segment : 64 time for calcul the mask position with numpy : 0.030459880828857422 nb_pixel_total : 1514987 time to create 1 rle with new method : 0.07724285125732422 length of segment : 1752 time for calcul the mask position with numpy : 0.00016045570373535156 nb_pixel_total : 5938 time to create 1 rle with old method : 0.0067217350006103516 length of segment : 130 time for calcul the mask position with numpy : 8.96453857421875e-05 nb_pixel_total : 3772 time to create 1 rle with old method : 0.0045049190521240234 length of segment : 101 time for calcul the mask position with numpy : 0.0001494884490966797 nb_pixel_total : 3764 time to create 1 rle with old method : 0.0046041011810302734 length of segment : 117 time for calcul the mask position with numpy : 0.00022983551025390625 nb_pixel_total : 16318 time to create 1 rle with old method : 0.0188748836517334 length of segment : 113 time for calcul the mask position with numpy : 0.0004634857177734375 nb_pixel_total : 30871 time to create 1 rle with old method : 0.03365349769592285 length of segment : 281 time for calcul the mask position with numpy : 0.00018453598022460938 nb_pixel_total : 9790 time to create 1 rle with old method : 0.010615825653076172 length of segment : 135 time for calcul the mask position with numpy : 0.0014252662658691406 nb_pixel_total : 92111 time to create 1 rle with old method : 0.0999748706817627 length of segment : 321 time for calcul the mask position with numpy : 0.005146026611328125 nb_pixel_total : 409272 time to create 1 rle with new method : 0.011765718460083008 length of segment : 543 time for calcul the mask position with numpy : 0.0012638568878173828 nb_pixel_total : 81884 time to create 1 rle with old method : 0.09018516540527344 length of segment : 356 time for calcul the mask position with numpy : 0.05902600288391113 nb_pixel_total : 4555861 time to create 1 rle with new method : 0.1326918601989746 length of segment : 3658 time for calcul the mask position with numpy : 0.001976490020751953 nb_pixel_total : 120229 time to create 1 rle with old method : 0.14298033714294434 length of segment : 447 time for calcul the mask position with numpy : 0.0014789104461669922 nb_pixel_total : 113691 time to create 1 rle with old method : 0.12277889251708984 length of segment : 464 time for calcul the mask position with numpy : 0.030364036560058594 nb_pixel_total : 1159019 time to create 1 rle with new method : 0.06465482711791992 length of segment : 2791 time for calcul the mask position with numpy : 0.0020751953125 nb_pixel_total : 162891 time to create 1 rle with new method : 0.0059185028076171875 length of segment : 541 time for calcul the mask position with numpy : 0.06703639030456543 nb_pixel_total : 2399887 time to create 1 rle with new method : 0.19819903373718262 length of segment : 3580 time for calcul the mask position with numpy : 0.0023834705352783203 nb_pixel_total : 69724 time to create 1 rle with old method : 0.10356712341308594 length of segment : 405 time for calcul the mask position with numpy : 0.004204273223876953 nb_pixel_total : 187080 time to create 1 rle with new method : 0.008021354675292969 length of segment : 804 time for calcul the mask position with numpy : 0.0009777545928955078 nb_pixel_total : 30083 time to create 1 rle with old method : 0.032881736755371094 length of segment : 304 time for calcul the mask position with numpy : 0.0005910396575927734 nb_pixel_total : 22152 time to create 1 rle with old method : 0.026441097259521484 length of segment : 172 time for calcul the mask position with numpy : 0.006169557571411133 nb_pixel_total : 340127 time to create 1 rle with new method : 0.010454893112182617 length of segment : 543 time for calcul the mask position with numpy : 0.005892038345336914 nb_pixel_total : 389491 time to create 1 rle with new method : 0.013859987258911133 length of segment : 625 time for calcul the mask position with numpy : 0.0003249645233154297 nb_pixel_total : 5091 time to create 1 rle with old method : 0.006474494934082031 length of segment : 70 time for calcul the mask position with numpy : 0.0005791187286376953 nb_pixel_total : 6522 time to create 1 rle with old method : 0.007887840270996094 length of segment : 175 time for calcul the mask position with numpy : 0.0005178451538085938 nb_pixel_total : 8834 time to create 1 rle with old method : 0.010571956634521484 length of segment : 179 time for calcul the mask position with numpy : 0.009793281555175781 nb_pixel_total : 310456 time to create 1 rle with new method : 0.012922048568725586 length of segment : 752 time for calcul the mask position with numpy : 0.001051187515258789 nb_pixel_total : 31113 time to create 1 rle with old method : 0.035748958587646484 length of segment : 260 time for calcul the mask position with numpy : 0.00281524658203125 nb_pixel_total : 88590 time to create 1 rle with old method : 0.10005950927734375 length of segment : 470 time for calcul the mask position with numpy : 0.0014984607696533203 nb_pixel_total : 33329 time to create 1 rle with old method : 0.03829836845397949 length of segment : 403 time for calcul the mask position with numpy : 0.0006988048553466797 nb_pixel_total : 9197 time to create 1 rle with old method : 0.010973215103149414 length of segment : 181 time for calcul the mask position with numpy : 0.0020134449005126953 nb_pixel_total : 45171 time to create 1 rle with old method : 0.05225014686584473 length of segment : 250 time for calcul the mask position with numpy : 0.010257720947265625 nb_pixel_total : 365426 time to create 1 rle with new method : 0.013403892517089844 length of segment : 581 time for calcul the mask position with numpy : 0.00047016143798828125 nb_pixel_total : 6666 time to create 1 rle with old method : 0.008471965789794922 length of segment : 131 time for calcul the mask position with numpy : 0.00011181831359863281 nb_pixel_total : 3522 time to create 1 rle with old method : 0.004359722137451172 length of segment : 75 time for calcul the mask position with numpy : 7.987022399902344e-05 nb_pixel_total : 2953 time to create 1 rle with old method : 0.003612518310546875 length of segment : 66 time for calcul the mask position with numpy : 0.002026796340942383 nb_pixel_total : 76578 time to create 1 rle with old method : 0.08865833282470703 length of segment : 303 time for calcul the mask position with numpy : 0.0015120506286621094 nb_pixel_total : 37406 time to create 1 rle with old method : 0.05470919609069824 length of segment : 217 time for calcul the mask position with numpy : 0.0017423629760742188 nb_pixel_total : 100965 time to create 1 rle with old method : 0.10933899879455566 length of segment : 417 time for calcul the mask position with numpy : 0.0004482269287109375 nb_pixel_total : 8237 time to create 1 rle with old method : 0.010291337966918945 length of segment : 128 time for calcul the mask position with numpy : 0.0005259513854980469 nb_pixel_total : 10001 time to create 1 rle with old method : 0.012067794799804688 length of segment : 156 time for calcul the mask position with numpy : 0.00037407875061035156 nb_pixel_total : 9224 time to create 1 rle with old method : 0.011194705963134766 length of segment : 112 time for calcul the mask position with numpy : 0.003013134002685547 nb_pixel_total : 122327 time to create 1 rle with old method : 0.13131952285766602 length of segment : 361 time for calcul the mask position with numpy : 0.0006012916564941406 nb_pixel_total : 15933 time to create 1 rle with old method : 0.019607067108154297 length of segment : 115 time for calcul the mask position with numpy : 0.0036764144897460938 nb_pixel_total : 79924 time to create 1 rle with old method : 0.08625292778015137 length of segment : 450 time for calcul the mask position with numpy : 0.001886606216430664 nb_pixel_total : 43318 time to create 1 rle with old method : 0.048809051513671875 length of segment : 455 time for calcul the mask position with numpy : 0.0009562969207763672 nb_pixel_total : 18865 time to create 1 rle with old method : 0.021512269973754883 length of segment : 287 time for calcul the mask position with numpy : 0.0008869171142578125 nb_pixel_total : 25057 time to create 1 rle with old method : 0.02883172035217285 length of segment : 170 time for calcul the mask position with numpy : 0.0023043155670166016 nb_pixel_total : 31388 time to create 1 rle with old method : 0.03429603576660156 length of segment : 604 time for calcul the mask position with numpy : 0.005427837371826172 nb_pixel_total : 117169 time to create 1 rle with old method : 0.15086746215820312 length of segment : 643 time for calcul the mask position with numpy : 0.002007007598876953 nb_pixel_total : 44264 time to create 1 rle with old method : 0.0545656681060791 length of segment : 417 time for calcul the mask position with numpy : 0.001982450485229492 nb_pixel_total : 60456 time to create 1 rle with old method : 0.06474161148071289 length of segment : 430 time for calcul the mask position with numpy : 0.00031638145446777344 nb_pixel_total : 3060 time to create 1 rle with old method : 0.0039825439453125 length of segment : 70 time for calcul the mask position with numpy : 0.0025072097778320312 nb_pixel_total : 64356 time to create 1 rle with old method : 0.0728752613067627 length of segment : 1094 time for calcul the mask position with numpy : 0.00020003318786621094 nb_pixel_total : 1821 time to create 1 rle with old method : 0.0023200511932373047 length of segment : 47 time for calcul the mask position with numpy : 0.02597975730895996 nb_pixel_total : 589773 time to create 1 rle with new method : 0.040282487869262695 length of segment : 1617 time for calcul the mask position with numpy : 0.0010840892791748047 nb_pixel_total : 35237 time to create 1 rle with old method : 0.038275957107543945 length of segment : 274 time for calcul the mask position with numpy : 0.007901668548583984 nb_pixel_total : 257288 time to create 1 rle with new method : 0.01284170150756836 length of segment : 690 time for calcul the mask position with numpy : 0.00019025802612304688 nb_pixel_total : 1945 time to create 1 rle with old method : 0.002417325973510742 length of segment : 57 time for calcul the mask position with numpy : 0.0004105567932128906 nb_pixel_total : 5457 time to create 1 rle with old method : 0.00709223747253418 length of segment : 148 time for calcul the mask position with numpy : 0.0001976490020751953 nb_pixel_total : 2004 time to create 1 rle with old method : 0.002551555633544922 length of segment : 49 time for calcul the mask position with numpy : 0.0009982585906982422 nb_pixel_total : 61141 time to create 1 rle with old method : 0.06721353530883789 length of segment : 428 time for calcul the mask position with numpy : 0.0001666545867919922 nb_pixel_total : 3651 time to create 1 rle with old method : 0.004535675048828125 length of segment : 108 time for calcul the mask position with numpy : 0.004263877868652344 nb_pixel_total : 165677 time to create 1 rle with new method : 0.012480497360229492 length of segment : 805 time for calcul the mask position with numpy : 0.003186464309692383 nb_pixel_total : 86826 time to create 1 rle with old method : 0.09475326538085938 length of segment : 446 time for calcul the mask position with numpy : 0.0001461505889892578 nb_pixel_total : 2624 time to create 1 rle with old method : 0.003344297409057617 length of segment : 69 time for calcul the mask position with numpy : 0.0027513504028320312 nb_pixel_total : 152422 time to create 1 rle with new method : 0.00609135627746582 length of segment : 453 time for calcul the mask position with numpy : 0.001142740249633789 nb_pixel_total : 21107 time to create 1 rle with old method : 0.0248260498046875 length of segment : 256 time for calcul the mask position with numpy : 0.006663322448730469 nb_pixel_total : 318235 time to create 1 rle with new method : 0.009641170501708984 length of segment : 398 time for calcul the mask position with numpy : 0.002818584442138672 nb_pixel_total : 159313 time to create 1 rle with new method : 0.00722050666809082 length of segment : 653 time for calcul the mask position with numpy : 0.000213623046875 nb_pixel_total : 6762 time to create 1 rle with old method : 0.00835418701171875 length of segment : 87 time for calcul the mask position with numpy : 0.0005793571472167969 nb_pixel_total : 7711 time to create 1 rle with old method : 0.009929418563842773 length of segment : 318 time for calcul the mask position with numpy : 0.0029654502868652344 nb_pixel_total : 87867 time to create 1 rle with old method : 0.10039067268371582 length of segment : 352 time for calcul the mask position with numpy : 0.0012505054473876953 nb_pixel_total : 27165 time to create 1 rle with old method : 0.03319668769836426 length of segment : 238 time for calcul the mask position with numpy : 0.0012447834014892578 nb_pixel_total : 32442 time to create 1 rle with old method : 0.03784298896789551 length of segment : 330 time for calcul the mask position with numpy : 0.0003676414489746094 nb_pixel_total : 5082 time to create 1 rle with old method : 0.006775379180908203 length of segment : 111 time for calcul the mask position with numpy : 0.0009214878082275391 nb_pixel_total : 17880 time to create 1 rle with old method : 0.02119588851928711 length of segment : 268 time for calcul the mask position with numpy : 0.0009672641754150391 nb_pixel_total : 31310 time to create 1 rle with old method : 0.03441500663757324 length of segment : 207 time for calcul the mask position with numpy : 0.0019996166229248047 nb_pixel_total : 28089 time to create 1 rle with old method : 0.03290224075317383 length of segment : 309 time for calcul the mask position with numpy : 0.00022268295288085938 nb_pixel_total : 3385 time to create 1 rle with old method : 0.0043375492095947266 length of segment : 44 time for calcul the mask position with numpy : 0.004471302032470703 nb_pixel_total : 163183 time to create 1 rle with new method : 0.006340980529785156 length of segment : 505 time for calcul the mask position with numpy : 0.0024328231811523438 nb_pixel_total : 77656 time to create 1 rle with old method : 0.08947014808654785 length of segment : 440 time for calcul the mask position with numpy : 0.003081083297729492 nb_pixel_total : 94117 time to create 1 rle with old method : 0.1057119369506836 length of segment : 334 time for calcul the mask position with numpy : 0.0005002021789550781 nb_pixel_total : 10177 time to create 1 rle with old method : 0.011486053466796875 length of segment : 131 time for calcul the mask position with numpy : 0.0013926029205322266 nb_pixel_total : 39775 time to create 1 rle with old method : 0.047446250915527344 length of segment : 316 time for calcul the mask position with numpy : 0.0003414154052734375 nb_pixel_total : 6486 time to create 1 rle with old method : 0.0074465274810791016 length of segment : 130 time for calcul the mask position with numpy : 0.000453948974609375 nb_pixel_total : 22929 time to create 1 rle with old method : 0.02965259552001953 length of segment : 269 time for calcul the mask position with numpy : 0.0003185272216796875 nb_pixel_total : 11044 time to create 1 rle with old method : 0.015078306198120117 length of segment : 139 time for calcul the mask position with numpy : 0.0002942085266113281 nb_pixel_total : 5675 time to create 1 rle with old method : 0.006557464599609375 length of segment : 80 time for calcul the mask position with numpy : 0.00434422492980957 nb_pixel_total : 145881 time to create 1 rle with old method : 0.16646790504455566 length of segment : 640 time for calcul the mask position with numpy : 0.0013463497161865234 nb_pixel_total : 31861 time to create 1 rle with old method : 0.038184165954589844 length of segment : 300 time for calcul the mask position with numpy : 0.0015714168548583984 nb_pixel_total : 54134 time to create 1 rle with old method : 0.059026479721069336 length of segment : 338 time for calcul the mask position with numpy : 0.00021505355834960938 nb_pixel_total : 3115 time to create 1 rle with old method : 0.003914833068847656 length of segment : 68 time for calcul the mask position with numpy : 0.0011150836944580078 nb_pixel_total : 32636 time to create 1 rle with old method : 0.03725934028625488 length of segment : 229 time for calcul the mask position with numpy : 0.00035858154296875 nb_pixel_total : 4952 time to create 1 rle with old method : 0.005815982818603516 length of segment : 147 time for calcul the mask position with numpy : 0.002366781234741211 nb_pixel_total : 69914 time to create 1 rle with old method : 0.08143758773803711 length of segment : 453 time for calcul the mask position with numpy : 0.0017919540405273438 nb_pixel_total : 41314 time to create 1 rle with old method : 0.0464780330657959 length of segment : 324 time for calcul the mask position with numpy : 0.0014431476593017578 nb_pixel_total : 50269 time to create 1 rle with old method : 0.05655527114868164 length of segment : 257 time for calcul the mask position with numpy : 0.0001239776611328125 nb_pixel_total : 2288 time to create 1 rle with old method : 0.0026504993438720703 length of segment : 64 time for calcul the mask position with numpy : 0.0006856918334960938 nb_pixel_total : 22267 time to create 1 rle with old method : 0.024686098098754883 length of segment : 134 time for calcul the mask position with numpy : 0.0003178119659423828 nb_pixel_total : 2608 time to create 1 rle with old method : 0.0033817291259765625 length of segment : 128 time for calcul the mask position with numpy : 0.0009126663208007812 nb_pixel_total : 17357 time to create 1 rle with old method : 0.020281076431274414 length of segment : 264 time for calcul the mask position with numpy : 0.0016231536865234375 nb_pixel_total : 50104 time to create 1 rle with old method : 0.05674552917480469 length of segment : 220 time for calcul the mask position with numpy : 0.0022962093353271484 nb_pixel_total : 79488 time to create 1 rle with old method : 0.08533859252929688 length of segment : 275 time for calcul the mask position with numpy : 0.005920886993408203 nb_pixel_total : 316542 time to create 1 rle with new method : 0.010419607162475586 length of segment : 597 time for calcul the mask position with numpy : 0.00033593177795410156 nb_pixel_total : 7566 time to create 1 rle with old method : 0.008096456527709961 length of segment : 92 time for calcul the mask position with numpy : 0.004245758056640625 nb_pixel_total : 164231 time to create 1 rle with new method : 0.006697893142700195 length of segment : 755 time for calcul the mask position with numpy : 0.00012135505676269531 nb_pixel_total : 1830 time to create 1 rle with old method : 0.003544330596923828 length of segment : 41 time for calcul the mask position with numpy : 0.0008938312530517578 nb_pixel_total : 34893 time to create 1 rle with old method : 0.053319454193115234 length of segment : 257 time for calcul the mask position with numpy : 0.0011448860168457031 nb_pixel_total : 19554 time to create 1 rle with old method : 0.02198314666748047 length of segment : 220 time for calcul the mask position with numpy : 0.003116607666015625 nb_pixel_total : 56982 time to create 1 rle with old method : 0.06334948539733887 length of segment : 340 time for calcul the mask position with numpy : 0.0005514621734619141 nb_pixel_total : 11964 time to create 1 rle with old method : 0.015350103378295898 length of segment : 125 time for calcul the mask position with numpy : 0.00023031234741210938 nb_pixel_total : 1857 time to create 1 rle with old method : 0.003020048141479492 length of segment : 44 time for calcul the mask position with numpy : 0.0014641284942626953 nb_pixel_total : 37293 time to create 1 rle with old method : 0.04293227195739746 length of segment : 265 time for calcul the mask position with numpy : 0.002350330352783203 nb_pixel_total : 55901 time to create 1 rle with old method : 0.06333374977111816 length of segment : 345 time for calcul the mask position with numpy : 0.0010156631469726562 nb_pixel_total : 31516 time to create 1 rle with old method : 0.03579115867614746 length of segment : 299 time for calcul the mask position with numpy : 0.006307363510131836 nb_pixel_total : 161691 time to create 1 rle with new method : 0.008413314819335938 length of segment : 907 time for calcul the mask position with numpy : 0.0006709098815917969 nb_pixel_total : 14334 time to create 1 rle with old method : 0.015471696853637695 length of segment : 158 time for calcul the mask position with numpy : 0.0021893978118896484 nb_pixel_total : 47208 time to create 1 rle with old method : 0.05631303787231445 length of segment : 415 time for calcul the mask position with numpy : 0.00022220611572265625 nb_pixel_total : 5747 time to create 1 rle with old method : 0.006925344467163086 length of segment : 67 time for calcul the mask position with numpy : 0.0016641616821289062 nb_pixel_total : 29987 time to create 1 rle with old method : 0.034334659576416016 length of segment : 199 time for calcul the mask position with numpy : 0.0022377967834472656 nb_pixel_total : 62465 time to create 1 rle with old method : 0.06616377830505371 length of segment : 340 time for calcul the mask position with numpy : 0.0008513927459716797 nb_pixel_total : 43480 time to create 1 rle with old method : 0.04746270179748535 length of segment : 306 time for calcul the mask position with numpy : 0.0010271072387695312 nb_pixel_total : 39462 time to create 1 rle with old method : 0.042936086654663086 length of segment : 276 time for calcul the mask position with numpy : 0.001871347427368164 nb_pixel_total : 55249 time to create 1 rle with old method : 0.057718515396118164 length of segment : 218 time for calcul the mask position with numpy : 0.0013589859008789062 nb_pixel_total : 22018 time to create 1 rle with old method : 0.028033018112182617 length of segment : 236 time for calcul the mask position with numpy : 0.0010211467742919922 nb_pixel_total : 18337 time to create 1 rle with old method : 0.020262718200683594 length of segment : 292 time for calcul the mask position with numpy : 0.00024580955505371094 nb_pixel_total : 6538 time to create 1 rle with old method : 0.007897377014160156 length of segment : 131 time for calcul the mask position with numpy : 0.0005092620849609375 nb_pixel_total : 12406 time to create 1 rle with old method : 0.01525259017944336 length of segment : 75 time for calcul the mask position with numpy : 0.0010039806365966797 nb_pixel_total : 23292 time to create 1 rle with old method : 0.026182889938354492 length of segment : 135 time for calcul the mask position with numpy : 0.0005271434783935547 nb_pixel_total : 10735 time to create 1 rle with old method : 0.01288461685180664 length of segment : 222 time for calcul the mask position with numpy : 0.00041031837463378906 nb_pixel_total : 22818 time to create 1 rle with old method : 0.02691030502319336 length of segment : 135 time for calcul the mask position with numpy : 0.00011873245239257812 nb_pixel_total : 1879 time to create 1 rle with old method : 0.0023365020751953125 length of segment : 53 time for calcul the mask position with numpy : 0.0007734298706054688 nb_pixel_total : 14418 time to create 1 rle with old method : 0.017064332962036133 length of segment : 154 time for calcul the mask position with numpy : 0.00013971328735351562 nb_pixel_total : 990 time to create 1 rle with old method : 0.0012254714965820312 length of segment : 35 time for calcul the mask position with numpy : 0.00018715858459472656 nb_pixel_total : 4501 time to create 1 rle with old method : 0.005398273468017578 length of segment : 112 time for calcul the mask position with numpy : 0.0024971961975097656 nb_pixel_total : 69966 time to create 1 rle with old method : 0.0757145881652832 length of segment : 385 time for calcul the mask position with numpy : 0.0018634796142578125 nb_pixel_total : 54035 time to create 1 rle with old method : 0.05748581886291504 length of segment : 331 time for calcul the mask position with numpy : 0.0010395050048828125 nb_pixel_total : 34420 time to create 1 rle with old method : 0.0397334098815918 length of segment : 296 time for calcul the mask position with numpy : 0.0003933906555175781 nb_pixel_total : 7438 time to create 1 rle with old method : 0.00868535041809082 length of segment : 226 time for calcul the mask position with numpy : 0.0020630359649658203 nb_pixel_total : 63680 time to create 1 rle with old method : 0.06755590438842773 length of segment : 343 time for calcul the mask position with numpy : 0.000339508056640625 nb_pixel_total : 4981 time to create 1 rle with old method : 0.005930185317993164 length of segment : 143 time for calcul the mask position with numpy : 0.0011844635009765625 nb_pixel_total : 35934 time to create 1 rle with old method : 0.03947186470031738 length of segment : 252 time for calcul the mask position with numpy : 0.0016164779663085938 nb_pixel_total : 55785 time to create 1 rle with old method : 0.06121110916137695 length of segment : 274 time for calcul the mask position with numpy : 0.0001423358917236328 nb_pixel_total : 2324 time to create 1 rle with old method : 0.0029764175415039062 length of segment : 66 time for calcul the mask position with numpy : 0.0010833740234375 nb_pixel_total : 36014 time to create 1 rle with old method : 0.04176902770996094 length of segment : 256 time for calcul the mask position with numpy : 0.00019288063049316406 nb_pixel_total : 2835 time to create 1 rle with old method : 0.003758668899536133 length of segment : 64 time for calcul the mask position with numpy : 0.0006995201110839844 nb_pixel_total : 12382 time to create 1 rle with old method : 0.013566017150878906 length of segment : 226 time for calcul the mask position with numpy : 6.818771362304688e-05 nb_pixel_total : 1395 time to create 1 rle with old method : 0.0017826557159423828 length of segment : 35 time for calcul the mask position with numpy : 0.006201744079589844 nb_pixel_total : 342668 time to create 1 rle with new method : 0.008576631546020508 length of segment : 519 time for calcul the mask position with numpy : 0.0007107257843017578 nb_pixel_total : 32974 time to create 1 rle with old method : 0.03787827491760254 length of segment : 230 time for calcul the mask position with numpy : 0.0007548332214355469 nb_pixel_total : 22182 time to create 1 rle with old method : 0.025173425674438477 length of segment : 134 time for calcul the mask position with numpy : 0.0019731521606445312 nb_pixel_total : 78693 time to create 1 rle with old method : 0.08397865295410156 length of segment : 273 time for calcul the mask position with numpy : 0.0016987323760986328 nb_pixel_total : 79700 time to create 1 rle with old method : 0.09276628494262695 length of segment : 278 time for calcul the mask position with numpy : 0.0011730194091796875 nb_pixel_total : 44071 time to create 1 rle with old method : 0.05050516128540039 length of segment : 214 time for calcul the mask position with numpy : 0.0003974437713623047 nb_pixel_total : 21596 time to create 1 rle with old method : 0.025070905685424805 length of segment : 130 time for calcul the mask position with numpy : 0.00036978721618652344 nb_pixel_total : 5524 time to create 1 rle with old method : 0.0065844058990478516 length of segment : 133 time for calcul the mask position with numpy : 0.0035936832427978516 nb_pixel_total : 129122 time to create 1 rle with old method : 0.1434941291809082 length of segment : 397 time for calcul the mask position with numpy : 0.0008590221405029297 nb_pixel_total : 22985 time to create 1 rle with old method : 0.025032997131347656 length of segment : 202 time for calcul the mask position with numpy : 0.00016164779663085938 nb_pixel_total : 1643 time to create 1 rle with old method : 0.002004384994506836 length of segment : 63 time for calcul the mask position with numpy : 0.0002498626708984375 nb_pixel_total : 2281 time to create 1 rle with old method : 0.0029823780059814453 length of segment : 88 time for calcul the mask position with numpy : 0.0009338855743408203 nb_pixel_total : 20370 time to create 1 rle with old method : 0.02660059928894043 length of segment : 135 time for calcul the mask position with numpy : 0.0007760524749755859 nb_pixel_total : 18289 time to create 1 rle with old method : 0.020764589309692383 length of segment : 182 time for calcul the mask position with numpy : 0.0029113292694091797 nb_pixel_total : 60283 time to create 1 rle with old method : 0.0676414966583252 length of segment : 572 time for calcul the mask position with numpy : 0.0048677921295166016 nb_pixel_total : 162832 time to create 1 rle with new method : 0.0064907073974609375 length of segment : 463 time for calcul the mask position with numpy : 0.0017969608306884766 nb_pixel_total : 40130 time to create 1 rle with old method : 0.04432320594787598 length of segment : 315 time for calcul the mask position with numpy : 0.0014791488647460938 nb_pixel_total : 39380 time to create 1 rle with old method : 0.04536747932434082 length of segment : 283 time for calcul the mask position with numpy : 0.0005812644958496094 nb_pixel_total : 18017 time to create 1 rle with old method : 0.02107524871826172 length of segment : 106 time for calcul the mask position with numpy : 0.0005159378051757812 nb_pixel_total : 20351 time to create 1 rle with old method : 0.023920059204101562 length of segment : 124 time for calcul the mask position with numpy : 0.0009119510650634766 nb_pixel_total : 5698 time to create 1 rle with old method : 0.007289886474609375 length of segment : 208 time for calcul the mask position with numpy : 0.00434565544128418 nb_pixel_total : 134570 time to create 1 rle with old method : 0.14578676223754883 length of segment : 632 time for calcul the mask position with numpy : 0.004746198654174805 nb_pixel_total : 186421 time to create 1 rle with new method : 0.0071904659271240234 length of segment : 578 time for calcul the mask position with numpy : 0.0015556812286376953 nb_pixel_total : 53806 time to create 1 rle with old method : 0.06322455406188965 length of segment : 318 time for calcul the mask position with numpy : 0.0012378692626953125 nb_pixel_total : 32955 time to create 1 rle with old method : 0.036375999450683594 length of segment : 299 time for calcul the mask position with numpy : 0.0019185543060302734 nb_pixel_total : 64365 time to create 1 rle with old method : 0.07385635375976562 length of segment : 349 time for calcul the mask position with numpy : 0.003264188766479492 nb_pixel_total : 104667 time to create 1 rle with old method : 0.1171410083770752 length of segment : 449 time for calcul the mask position with numpy : 0.0013489723205566406 nb_pixel_total : 60008 time to create 1 rle with old method : 0.07043623924255371 length of segment : 426 time for calcul the mask position with numpy : 0.00041365623474121094 nb_pixel_total : 20096 time to create 1 rle with old method : 0.024140357971191406 length of segment : 117 time for calcul the mask position with numpy : 0.002022981643676758 nb_pixel_total : 64279 time to create 1 rle with old method : 0.07255125045776367 length of segment : 501 time spent for convertir_results : 18.776623964309692 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 182 chid ids of type : 3663 Number RLEs to save : 63807 save missing photos in datou_result : time spend for datou_step_exec : 67.72519874572754 time spend to save output : 7.432122468948364 total time spend for step 1 : 75.1573212146759 step2:crop_condition Fri Feb 7 01:33:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 182 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 22 About to insert : list_path_to_insert length 22 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 13 About to insert : list_path_to_insert length 13 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 60 About to insert : list_path_to_insert length 60 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335314346, 1335314324, 1335314322, 1335314302, 1335314295, 1335314287, 1335314283, 1335314281, 1335314278, 1335314182] Looping around the photos to save general results len do output : 119 /-3658499628Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499682Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499688Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499681Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499703Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499705Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499709Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499723Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499722Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499721Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499750Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499741Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499746Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499777Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499768Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499769Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499789Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499798Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499803Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499795Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499793Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499800Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499652Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499659Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499679Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499663Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499698Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499693Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499695Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499700Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499715Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499764Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499796Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499801Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499794Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499639Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499643Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499669Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499712Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499727Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499745Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499776Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499808Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499641Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499644Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499651Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499649Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499668Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499680Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499675Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499667Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499657Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499673Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499671Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499672Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499677Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499661Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499676Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499658Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499684Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499699Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499697Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499694Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499702Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499687Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499713Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499704Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499718Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499711Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499717Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499708Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499732Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499737Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499726Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499734Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499733Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499724Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499740Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499735Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499749Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499751Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499757Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499747Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499767Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499752Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499754Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499758Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499755Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499744Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499756Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499778Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499771Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499784Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499785Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499773Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499780Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499799Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499788Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499804Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499806Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499807Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499792Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499805Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499631Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499630Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499662Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499674Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499692Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499730Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499753Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499770Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499637Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499650Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499646Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499728Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499774Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499714Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499760Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658499782Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314346', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314324', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314322', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314302', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314295', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314287', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314283', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314281', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314278', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314182', None, None, None, None, None, '2560904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 367 time used for this insertion : 0.03744840621948242 save_final save missing photos in datou_result : time spend for datou_step_exec : 44.62547588348389 time spend to save output : 0.0407259464263916 total time spend for step 2 : 44.66620182991028 step3:thcl Fri Feb 7 01:34:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.03694605827331543 time to convert the images to numpy array : 0.13440990447998047 time to import caffe and check if the image exist : 0.03063488006591797 time to convert the images to numpy array : 0.1402754783630371 time to import caffe and check if the image exist : 0.03763914108276367 time to convert the images to numpy array : 0.1470167636871338 time to import caffe and check if the image exist : 0.030590057373046875 time to convert the images to numpy array : 0.17079520225524902 time to import caffe and check if the image exist : 0.032976388931274414 time to convert the images to numpy array : 0.16866087913513184 time to import caffe and check if the image exist : 0.03016066551208496 time to convert the images to numpy array : 0.17301011085510254 time to import caffe and check if the image exist : 0.031874895095825195 time to convert the images to numpy array : 0.18306589126586914 time to import caffe and check if the image exist : 0.036646366119384766 time to convert the images to numpy array : 0.17950868606567383 time to import caffe and check if the image exist : 0.033967018127441406 time to convert the images to numpy array : 0.18495798110961914 time to import caffe and check if the image exist : 0.030511856079101562 time to convert the images to numpy array : 0.18977570533752441 total time to convert the images to numpy array : 1.2614006996154785 list photo_ids error: [] list photo_ids correct : [-3658499628, -3658499682, -3658499688, -3658499681, -3658499703, -3658499705, -3658499709, -3658499723, -3658499722, -3658499721, -3658499750, -3658499741, -3658499799, -3658499788, -3658499804, -3658499806, -3658499807, -3658499792, -3658499805, -3658499631, -3658499630, -3658499662, -3658499674, -3658499692, -3658499730, -3658499753, -3658499770, -3658499637, -3658499650, -3658499646, -3658499728, -3658499774, -3658499714, -3658499760, -3658499782, -3658499680, -3658499675, -3658499667, -3658499657, -3658499673, -3658499671, -3658499672, -3658499677, -3658499661, -3658499676, -3658499658, -3658499684, -3658499752, -3658499754, -3658499758, -3658499755, -3658499744, -3658499756, -3658499778, -3658499771, -3658499784, -3658499785, -3658499773, -3658499780, -3658499746, -3658499777, -3658499768, -3658499769, -3658499789, -3658499798, -3658499803, -3658499795, -3658499793, -3658499800, -3658499652, -3658499659, -3658499679, -3658499663, -3658499698, -3658499693, -3658499695, -3658499700, -3658499715, -3658499764, -3658499796, -3658499801, -3658499794, -3658499639, -3658499699, -3658499697, -3658499694, -3658499702, -3658499687, -3658499713, -3658499704, -3658499718, -3658499711, -3658499717, -3658499708, -3658499732, -3658499737, -3658499726, -3658499734, -3658499733, -3658499724, -3658499740, -3658499735, -3658499749, -3658499751, -3658499757, -3658499747, -3658499767, -3658499643, -3658499669, -3658499712, -3658499727, -3658499745, -3658499776, -3658499808, -3658499641, -3658499644, -3658499651, -3658499649, -3658499668] number of photos to traite : 119 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10772 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10553 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.171701431274414 time used to do the prediction : 0.32332777976989746 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10270 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 9752 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.2283124923706055 time used to do the prediction : 0.20480918884277344 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.5497207641601562e-05 save missing photos in datou_result : time spend for datou_step_exec : 12.806058883666992 time spend to save output : 0.0008106231689453125 total time spend for step 3 : 12.806869506835938 step4:merge_mask_thcl_custom Fri Feb 7 01:34:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 182 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1335314346, 1335314324, 1335314322, 1335314302, 1335314295, 1335314287, 1335314283, 1335314281, 1335314278, 1335314182] Looping around the photos to save general results len do output : 10 /1335314346Didn't retrieve data .Didn't retrieve data . /1335314324Didn't retrieve data .Didn't retrieve data . /1335314322Didn't retrieve data .Didn't retrieve data . /1335314302Didn't retrieve data .Didn't retrieve data . /1335314295Didn't retrieve data .Didn't retrieve data . /1335314287Didn't retrieve data .Didn't retrieve data . /1335314283Didn't retrieve data .Didn't retrieve data . /1335314281Didn't retrieve data .Didn't retrieve data . /1335314278Didn't retrieve data .Didn't retrieve data . /1335314182Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314346', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314324', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314322', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314302', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314295', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314287', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314283', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314281', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314278', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314182', None, None, None, None, None, '2560904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014658689498901367 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.06869697570800781 time spend to save output : 0.01524972915649414 total time spend for step 4 : 0.08394670486450195 step5:rle_unique_nms_with_priority Fri Feb 7 01:34:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 13 nb_hashtags : 5 time to prepare the origin masks : 8.96976900100708 time for calcul the mask position with numpy : 0.22610783576965332 nb_pixel_total : 4719411 time to create 1 rle with new method : 0.6575250625610352 time for calcul the mask position with numpy : 0.026203155517578125 nb_pixel_total : 30859 time to create 1 rle with old method : 0.03598785400390625 time for calcul the mask position with numpy : 0.024319887161254883 nb_pixel_total : 408164 time to create 1 rle with new method : 0.1362621784210205 time for calcul the mask position with numpy : 0.021627426147460938 nb_pixel_total : 41238 time to create 1 rle with old method : 0.045139312744140625 time for calcul the mask position with numpy : 0.02196788787841797 nb_pixel_total : 113425 time to create 1 rle with old method : 0.11978721618652344 time for calcul the mask position with numpy : 0.037203311920166016 nb_pixel_total : 16271 time to create 1 rle with old method : 0.017697572708129883 time for calcul the mask position with numpy : 0.033161163330078125 nb_pixel_total : 3710 time to create 1 rle with old method : 0.004312753677368164 time for calcul the mask position with numpy : 0.0294191837310791 nb_pixel_total : 3752 time to create 1 rle with old method : 0.00429081916809082 time for calcul the mask position with numpy : 0.04196786880493164 nb_pixel_total : 3404 time to create 1 rle with old method : 0.004099845886230469 time for calcul the mask position with numpy : 0.03624415397644043 nb_pixel_total : 3179 time to create 1 rle with old method : 0.0036754608154296875 time for calcul the mask position with numpy : 0.04152846336364746 nb_pixel_total : 14826 time to create 1 rle with old method : 0.017541885375976562 time for calcul the mask position with numpy : 0.05740714073181152 nb_pixel_total : 1513669 time to create 1 rle with new method : 0.6802947521209717 time for calcul the mask position with numpy : 0.03631424903869629 nb_pixel_total : 96617 time to create 1 rle with old method : 0.11780452728271484 time for calcul the mask position with numpy : 0.030847549438476562 nb_pixel_total : 81715 time to create 1 rle with old method : 0.08708333969116211 create new chi : 2.6647002696990967 time to delete rle : 0.010635852813720703 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 11434 TO DO : save crop sub photo not yet done ! save time : 1.0342695713043213 nb_obj : 3 nb_hashtags : 3 time to prepare the origin masks : 1.796638011932373 time for calcul the mask position with numpy : 0.03322577476501465 nb_pixel_total : 1915362 time to create 1 rle with new method : 0.19506049156188965 time for calcul the mask position with numpy : 0.021714448928833008 nb_pixel_total : 119946 time to create 1 rle with old method : 0.12989425659179688 time for calcul the mask position with numpy : 0.02612161636352539 nb_pixel_total : 113707 time to create 1 rle with old method : 0.13000941276550293 time for calcul the mask position with numpy : 0.5748724937438965 nb_pixel_total : 4901225 time to create 1 rle with new method : 0.8371906280517578 create new chi : 2.0074145793914795 time to delete rle : 0.0013737678527832031 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 9106 TO DO : save crop sub photo not yet done ! save time : 0.5300874710083008 nb_obj : 9 nb_hashtags : 5 time to prepare the origin masks : 6.2246763706207275 time for calcul the mask position with numpy : 0.3064579963684082 nb_pixel_total : 3465174 time to create 1 rle with new method : 0.7052276134490967 time for calcul the mask position with numpy : 0.023773670196533203 nb_pixel_total : 22910 time to create 1 rle with old method : 0.025726318359375 time for calcul the mask position with numpy : 0.022662878036499023 nb_pixel_total : 22093 time to create 1 rle with old method : 0.02356123924255371 time for calcul the mask position with numpy : 0.02417445182800293 nb_pixel_total : 282895 time to create 1 rle with new method : 0.14758682250976562 time for calcul the mask position with numpy : 0.037273406982421875 nb_pixel_total : 30023 time to create 1 rle with old method : 0.03279399871826172 time for calcul the mask position with numpy : 0.03738236427307129 nb_pixel_total : 70422 time to create 1 rle with old method : 0.07926344871520996 time for calcul the mask position with numpy : 0.039746999740600586 nb_pixel_total : 69449 time to create 1 rle with old method : 0.07761096954345703 time for calcul the mask position with numpy : 0.0536649227142334 nb_pixel_total : 1787674 time to create 1 rle with new method : 0.7732601165771484 time for calcul the mask position with numpy : 0.0506131649017334 nb_pixel_total : 1136941 time to create 1 rle with new method : 0.9341425895690918 time for calcul the mask position with numpy : 0.029770612716674805 nb_pixel_total : 162659 time to create 1 rle with new method : 0.9485418796539307 create new chi : 4.485391855239868 time to delete rle : 0.0024890899658203125 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 19016 TO DO : save crop sub photo not yet done ! save time : 2.221355438232422 nb_obj : 27 nb_hashtags : 7 time to prepare the origin masks : 9.705934524536133 time for calcul the mask position with numpy : 0.0776364803314209 nb_pixel_total : 5544198 time to create 1 rle with new method : 0.14708733558654785 time for calcul the mask position with numpy : 0.02899479866027832 nb_pixel_total : 6615 time to create 1 rle with old method : 0.007414817810058594 time for calcul the mask position with numpy : 0.028860092163085938 nb_pixel_total : 6486 time to create 1 rle with old method : 0.0073893070220947266 time for calcul the mask position with numpy : 0.028676748275756836 nb_pixel_total : 8803 time to create 1 rle with old method : 0.009957551956176758 time for calcul the mask position with numpy : 0.03032207489013672 nb_pixel_total : 88336 time to create 1 rle with old method : 0.09821653366088867 time for calcul the mask position with numpy : 0.034089088439941406 nb_pixel_total : 310290 time to create 1 rle with new method : 0.1257319450378418 time for calcul the mask position with numpy : 0.028815031051635742 nb_pixel_total : 9145 time to create 1 rle with old method : 0.010267972946166992 time for calcul the mask position with numpy : 0.028961658477783203 nb_pixel_total : 15896 time to create 1 rle with old method : 0.01687145233154297 time for calcul the mask position with numpy : 0.029032468795776367 nb_pixel_total : 79821 time to create 1 rle with old method : 0.0878288745880127 time for calcul the mask position with numpy : 0.028992891311645508 nb_pixel_total : 29483 time to create 1 rle with old method : 0.0314793586730957 time for calcul the mask position with numpy : 0.028887033462524414 nb_pixel_total : 2941 time to create 1 rle with old method : 0.003346681594848633 time for calcul the mask position with numpy : 0.028821229934692383 nb_pixel_total : 18750 time to create 1 rle with old method : 0.02055954933166504 time for calcul the mask position with numpy : 0.028843164443969727 nb_pixel_total : 8154 time to create 1 rle with old method : 0.020337343215942383 time for calcul the mask position with numpy : 0.02985358238220215 nb_pixel_total : 100587 time to create 1 rle with old method : 0.10663461685180664 time for calcul the mask position with numpy : 0.028998374938964844 nb_pixel_total : 2460 time to create 1 rle with old method : 0.003141641616821289 time for calcul the mask position with numpy : 0.029375314712524414 nb_pixel_total : 3365 time to create 1 rle with old method : 0.003876209259033203 time for calcul the mask position with numpy : 0.030734777450561523 nb_pixel_total : 122013 time to create 1 rle with old method : 0.19254183769226074 time for calcul the mask position with numpy : 0.038373708724975586 nb_pixel_total : 37297 time to create 1 rle with old method : 0.05582404136657715 time for calcul the mask position with numpy : 0.03213620185852051 nb_pixel_total : 33412 time to create 1 rle with old method : 0.03607535362243652 time for calcul the mask position with numpy : 0.0305025577545166 nb_pixel_total : 14981 time to create 1 rle with old method : 0.016420841217041016 time for calcul the mask position with numpy : 0.034210205078125 nb_pixel_total : 116923 time to create 1 rle with old method : 0.13107085227966309 time for calcul the mask position with numpy : 0.030778884887695312 nb_pixel_total : 31299 time to create 1 rle with old method : 0.05100250244140625 time for calcul the mask position with numpy : 0.04213404655456543 nb_pixel_total : 364669 time to create 1 rle with new method : 0.15477776527404785 time for calcul the mask position with numpy : 0.03489208221435547 nb_pixel_total : 44985 time to create 1 rle with old method : 0.06798005104064941 time for calcul the mask position with numpy : 0.0341191291809082 nb_pixel_total : 9204 time to create 1 rle with old method : 0.01440286636352539 time for calcul the mask position with numpy : 0.03548550605773926 nb_pixel_total : 10179 time to create 1 rle with old method : 0.016060590744018555 time for calcul the mask position with numpy : 0.02969837188720703 nb_pixel_total : 24917 time to create 1 rle with old method : 0.026525497436523438 time for calcul the mask position with numpy : 0.029055356979370117 nb_pixel_total : 5031 time to create 1 rle with old method : 0.005481243133544922 create new chi : 2.432192802429199 time to delete rle : 0.0017290115356445312 batch 1 Loaded 28 chid ids of type : 3726 Number RLEs to save : 17310 TO DO : save crop sub photo not yet done ! save time : 0.9868321418762207 nb_obj : 22 nb_hashtags : 5 time to prepare the origin masks : 12.43384075164795 time for calcul the mask position with numpy : 0.16776728630065918 nb_pixel_total : 5351685 time to create 1 rle with new method : 0.7449448108673096 time for calcul the mask position with numpy : 0.022614240646362305 nb_pixel_total : 67941 time to create 1 rle with old method : 0.08919930458068848 time for calcul the mask position with numpy : 0.021394729614257812 nb_pixel_total : 14797 time to create 1 rle with old method : 0.016298770904541016 time for calcul the mask position with numpy : 0.021645545959472656 nb_pixel_total : 43722 time to create 1 rle with old method : 0.04698038101196289 time for calcul the mask position with numpy : 0.021225929260253906 nb_pixel_total : 2120 time to create 1 rle with old method : 0.002717733383178711 time for calcul the mask position with numpy : 0.02563786506652832 nb_pixel_total : 6708 time to create 1 rle with old method : 0.007401943206787109 time for calcul the mask position with numpy : 0.03221273422241211 nb_pixel_total : 60419 time to create 1 rle with old method : 0.06407976150512695 time for calcul the mask position with numpy : 0.034886837005615234 nb_pixel_total : 14356 time to create 1 rle with old method : 0.015816926956176758 time for calcul the mask position with numpy : 0.020803213119506836 nb_pixel_total : 2030 time to create 1 rle with old method : 0.0022258758544921875 time for calcul the mask position with numpy : 0.02313375473022461 nb_pixel_total : 317037 time to create 1 rle with new method : 0.29641079902648926 time for calcul the mask position with numpy : 0.02063775062561035 nb_pixel_total : 3636 time to create 1 rle with old method : 0.004175901412963867 time for calcul the mask position with numpy : 0.02151775360107422 nb_pixel_total : 2004 time to create 1 rle with old method : 0.0023097991943359375 time for calcul the mask position with numpy : 0.02131938934326172 nb_pixel_total : 5425 time to create 1 rle with old method : 0.006170034408569336 time for calcul the mask position with numpy : 0.021146774291992188 nb_pixel_total : 35161 time to create 1 rle with old method : 0.03743910789489746 time for calcul the mask position with numpy : 0.02176952362060547 nb_pixel_total : 1937 time to create 1 rle with old method : 0.0021212100982666016 time for calcul the mask position with numpy : 0.027568817138671875 nb_pixel_total : 588526 time to create 1 rle with new method : 0.3124856948852539 time for calcul the mask position with numpy : 0.0213623046875 nb_pixel_total : 1812 time to create 1 rle with old method : 0.0020949840545654297 time for calcul the mask position with numpy : 0.02156233787536621 nb_pixel_total : 3025 time to create 1 rle with old method : 0.003493785858154297 time for calcul the mask position with numpy : 0.021579742431640625 nb_pixel_total : 24253 time to create 1 rle with old method : 0.026072978973388672 time for calcul the mask position with numpy : 0.02356886863708496 nb_pixel_total : 152153 time to create 1 rle with new method : 0.2778048515319824 time for calcul the mask position with numpy : 0.02178215980529785 nb_pixel_total : 86794 time to create 1 rle with old method : 0.09203147888183594 time for calcul the mask position with numpy : 0.023438453674316406 nb_pixel_total : 257015 time to create 1 rle with new method : 0.4662644863128662 time for calcul the mask position with numpy : 0.026743173599243164 nb_pixel_total : 7684 time to create 1 rle with old method : 0.008372068405151367 create new chi : 3.313549041748047 time to delete rle : 0.0015234947204589844 batch 1 Loaded 23 chid ids of type : 3726 Number RLEs to save : 16966 TO DO : save crop sub photo not yet done ! save time : 2.2646379470825195 nb_obj : 17 nb_hashtags : 5 time to prepare the origin masks : 7.054574966430664 time for calcul the mask position with numpy : 0.508974552154541 nb_pixel_total : 6286065 time to create 1 rle with new method : 0.9795427322387695 time for calcul the mask position with numpy : 0.02240300178527832 nb_pixel_total : 87696 time to create 1 rle with old method : 0.09696841239929199 time for calcul the mask position with numpy : 0.0224149227142334 nb_pixel_total : 145578 time to create 1 rle with old method : 0.16240692138671875 time for calcul the mask position with numpy : 0.020830631256103516 nb_pixel_total : 22868 time to create 1 rle with old method : 0.024363279342651367 time for calcul the mask position with numpy : 0.020937681198120117 nb_pixel_total : 6475 time to create 1 rle with old method : 0.0073528289794921875 time for calcul the mask position with numpy : 0.020929336547851562 nb_pixel_total : 245 time to create 1 rle with old method : 0.0005426406860351562 time for calcul the mask position with numpy : 0.024170398712158203 nb_pixel_total : 3356 time to create 1 rle with old method : 0.0038928985595703125 time for calcul the mask position with numpy : 0.028379440307617188 nb_pixel_total : 93586 time to create 1 rle with old method : 0.09865570068359375 time for calcul the mask position with numpy : 0.020950794219970703 nb_pixel_total : 167 time to create 1 rle with old method : 0.0002875328063964844 time for calcul the mask position with numpy : 0.032297611236572266 nb_pixel_total : 27130 time to create 1 rle with old method : 0.03036046028137207 time for calcul the mask position with numpy : 0.03478384017944336 nb_pixel_total : 17878 time to create 1 rle with old method : 0.019002676010131836 time for calcul the mask position with numpy : 0.03418087959289551 nb_pixel_total : 4984 time to create 1 rle with old method : 0.0058078765869140625 time for calcul the mask position with numpy : 0.03540754318237305 nb_pixel_total : 32256 time to create 1 rle with old method : 0.03438091278076172 time for calcul the mask position with numpy : 0.0380244255065918 nb_pixel_total : 11002 time to create 1 rle with old method : 0.012357950210571289 time for calcul the mask position with numpy : 0.03522539138793945 nb_pixel_total : 77182 time to create 1 rle with old method : 0.0820772647857666 time for calcul the mask position with numpy : 0.03466057777404785 nb_pixel_total : 31318 time to create 1 rle with old method : 0.03730154037475586 time for calcul the mask position with numpy : 0.03840756416320801 nb_pixel_total : 39609 time to create 1 rle with old method : 0.05165839195251465 time for calcul the mask position with numpy : 0.03726792335510254 nb_pixel_total : 162845 time to create 1 rle with new method : 1.1409871578216553 create new chi : 3.854888439178467 time to delete rle : 0.0017125606536865234 batch 1 Loaded 18 chid ids of type : 3726 Number RLEs to save : 10955 TO DO : save crop sub photo not yet done ! save time : 1.6937661170959473 nb_obj : 19 nb_hashtags : 8 time to prepare the origin masks : 10.668267250061035 time for calcul the mask position with numpy : 1.0043461322784424 nb_pixel_total : 6082889 time to create 1 rle with new method : 0.8650195598602295 time for calcul the mask position with numpy : 0.03152608871459961 nb_pixel_total : 79147 time to create 1 rle with old method : 0.08454084396362305 time for calcul the mask position with numpy : 0.030248641967773438 nb_pixel_total : 50009 time to create 1 rle with old method : 0.052646636962890625 time for calcul the mask position with numpy : 0.02899765968322754 nb_pixel_total : 19495 time to create 1 rle with old method : 0.021626949310302734 time for calcul the mask position with numpy : 0.0324554443359375 nb_pixel_total : 1028 time to create 1 rle with old method : 0.0013120174407958984 time for calcul the mask position with numpy : 0.02955913543701172 nb_pixel_total : 3065 time to create 1 rle with old method : 0.003661632537841797 time for calcul the mask position with numpy : 0.04159975051879883 nb_pixel_total : 22136 time to create 1 rle with old method : 0.024249553680419922 time for calcul the mask position with numpy : 0.052077531814575195 nb_pixel_total : 7495 time to create 1 rle with old method : 0.008098602294921875 time for calcul the mask position with numpy : 0.04193377494812012 nb_pixel_total : 316103 time to create 1 rle with new method : 0.8006322383880615 time for calcul the mask position with numpy : 0.03495955467224121 nb_pixel_total : 17319 time to create 1 rle with old method : 0.01965951919555664 time for calcul the mask position with numpy : 0.03666543960571289 nb_pixel_total : 2603 time to create 1 rle with old method : 0.0030450820922851562 time for calcul the mask position with numpy : 0.037841796875 nb_pixel_total : 32574 time to create 1 rle with old method : 0.0347745418548584 time for calcul the mask position with numpy : 0.03496694564819336 nb_pixel_total : 54090 time to create 1 rle with old method : 0.057060956954956055 time for calcul the mask position with numpy : 0.0351109504699707 nb_pixel_total : 2277 time to create 1 rle with old method : 0.0025849342346191406 time for calcul the mask position with numpy : 0.035790205001831055 nb_pixel_total : 50144 time to create 1 rle with old method : 0.0532991886138916 time for calcul the mask position with numpy : 0.03414773941040039 nb_pixel_total : 41087 time to create 1 rle with old method : 0.04353070259094238 time for calcul the mask position with numpy : 0.03261446952819824 nb_pixel_total : 162380 time to create 1 rle with new method : 0.6694984436035156 time for calcul the mask position with numpy : 0.029448270797729492 nb_pixel_total : 4932 time to create 1 rle with old method : 0.005637407302856445 time for calcul the mask position with numpy : 0.022571086883544922 nb_pixel_total : 31754 time to create 1 rle with old method : 0.035774946212768555 time for calcul the mask position with numpy : 0.035138845443725586 nb_pixel_total : 69713 time to create 1 rle with old method : 0.07463455200195312 create new chi : 4.631123781204224 time to delete rle : 0.0014920234680175781 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 11935 TO DO : save crop sub photo not yet done ! save time : 1.7131679058074951 nb_obj : 25 nb_hashtags : 7 time to prepare the origin masks : 14.711678981781006 time for calcul the mask position with numpy : 0.3185861110687256 nb_pixel_total : 6306501 time to create 1 rle with new method : 0.8698310852050781 time for calcul the mask position with numpy : 0.02563786506652832 nb_pixel_total : 55115 time to create 1 rle with old method : 0.058602094650268555 time for calcul the mask position with numpy : 0.025635957717895508 nb_pixel_total : 56813 time to create 1 rle with old method : 0.06240487098693848 time for calcul the mask position with numpy : 0.0257875919342041 nb_pixel_total : 37213 time to create 1 rle with old method : 0.039943695068359375 time for calcul the mask position with numpy : 0.026477336883544922 nb_pixel_total : 940 time to create 1 rle with old method : 0.0013628005981445312 time for calcul the mask position with numpy : 0.02254033088684082 nb_pixel_total : 65 time to create 1 rle with old method : 0.0002646446228027344 time for calcul the mask position with numpy : 0.03990983963012695 nb_pixel_total : 3663 time to create 1 rle with old method : 0.005831241607666016 time for calcul the mask position with numpy : 0.04034018516540527 nb_pixel_total : 980 time to create 1 rle with old method : 0.0011451244354248047 time for calcul the mask position with numpy : 0.0361328125 nb_pixel_total : 1862 time to create 1 rle with old method : 0.002154111862182617 time for calcul the mask position with numpy : 0.038350582122802734 nb_pixel_total : 69828 time to create 1 rle with old method : 0.07383489608764648 time for calcul the mask position with numpy : 0.03608250617980957 nb_pixel_total : 55628 time to create 1 rle with old method : 0.060115814208984375 time for calcul the mask position with numpy : 0.03528642654418945 nb_pixel_total : 31440 time to create 1 rle with old method : 0.0352480411529541 time for calcul the mask position with numpy : 0.037325382232666016 nb_pixel_total : 12301 time to create 1 rle with old method : 0.013791322708129883 time for calcul the mask position with numpy : 0.03561878204345703 nb_pixel_total : 14323 time to create 1 rle with old method : 0.016713619232177734 time for calcul the mask position with numpy : 0.03759431838989258 nb_pixel_total : 23202 time to create 1 rle with old method : 0.025303363800048828 time for calcul the mask position with numpy : 0.03735685348510742 nb_pixel_total : 47152 time to create 1 rle with old method : 0.06882548332214355 time for calcul the mask position with numpy : 0.03800320625305176 nb_pixel_total : 21996 time to create 1 rle with old method : 0.023605823516845703 time for calcul the mask position with numpy : 0.036926984786987305 nb_pixel_total : 5191 time to create 1 rle with old method : 0.005616426467895508 time for calcul the mask position with numpy : 0.0374298095703125 nb_pixel_total : 29755 time to create 1 rle with old method : 0.03180575370788574 time for calcul the mask position with numpy : 0.03930091857910156 nb_pixel_total : 161127 time to create 1 rle with new method : 0.4233365058898926 time for calcul the mask position with numpy : 0.03596138954162598 nb_pixel_total : 18348 time to create 1 rle with old method : 0.01992058753967285 time for calcul the mask position with numpy : 0.037996768951416016 nb_pixel_total : 14181 time to create 1 rle with old method : 0.015871763229370117 time for calcul the mask position with numpy : 0.03813791275024414 nb_pixel_total : 6478 time to create 1 rle with old method : 0.0071849822998046875 time for calcul the mask position with numpy : 0.03553438186645508 nb_pixel_total : 1845 time to create 1 rle with old method : 0.0022020339965820312 time for calcul the mask position with numpy : 0.047063350677490234 nb_pixel_total : 11835 time to create 1 rle with old method : 0.03318381309509277 time for calcul the mask position with numpy : 0.036348581314086914 nb_pixel_total : 62458 time to create 1 rle with old method : 0.0691986083984375 create new chi : 3.240114450454712 time to delete rle : 0.0017066001892089844 batch 1 Loaded 26 chid ids of type : 3726 Number RLEs to save : 13008 TO DO : save crop sub photo not yet done ! save time : 1.8516802787780762 nb_obj : 16 nb_hashtags : 8 time to prepare the origin masks : 9.429742336273193 time for calcul the mask position with numpy : 0.4296834468841553 nb_pixel_total : 6252570 time to create 1 rle with new method : 0.6411077976226807 time for calcul the mask position with numpy : 0.031148433685302734 nb_pixel_total : 342300 time to create 1 rle with new method : 0.7703275680541992 time for calcul the mask position with numpy : 0.030150413513183594 nb_pixel_total : 43908 time to create 1 rle with old method : 0.04694843292236328 time for calcul the mask position with numpy : 0.030334949493408203 nb_pixel_total : 35892 time to create 1 rle with old method : 0.038280487060546875 time for calcul the mask position with numpy : 0.05482935905456543 nb_pixel_total : 2479 time to create 1 rle with old method : 0.0029213428497314453 time for calcul the mask position with numpy : 0.04355049133300781 nb_pixel_total : 1389 time to create 1 rle with old method : 0.0017557144165039062 time for calcul the mask position with numpy : 0.05453610420227051 nb_pixel_total : 22049 time to create 1 rle with old method : 0.02474045753479004 time for calcul the mask position with numpy : 0.04291987419128418 nb_pixel_total : 79376 time to create 1 rle with old method : 0.08594107627868652 time for calcul the mask position with numpy : 0.03734898567199707 nb_pixel_total : 7376 time to create 1 rle with old method : 0.008571147918701172 time for calcul the mask position with numpy : 0.05191469192504883 nb_pixel_total : 34409 time to create 1 rle with old method : 0.038033246994018555 time for calcul the mask position with numpy : 0.0415036678314209 nb_pixel_total : 55638 time to create 1 rle with old method : 0.05766654014587402 time for calcul the mask position with numpy : 0.04473114013671875 nb_pixel_total : 2318 time to create 1 rle with old method : 0.002809762954711914 time for calcul the mask position with numpy : 0.04170417785644531 nb_pixel_total : 12343 time to create 1 rle with old method : 0.01352071762084961 time for calcul the mask position with numpy : 0.040845394134521484 nb_pixel_total : 35892 time to create 1 rle with old method : 0.04031252861022949 time for calcul the mask position with numpy : 0.04401111602783203 nb_pixel_total : 4975 time to create 1 rle with old method : 0.005501747131347656 time for calcul the mask position with numpy : 0.03810882568359375 nb_pixel_total : 53867 time to create 1 rle with old method : 0.057547569274902344 time for calcul the mask position with numpy : 0.04096031188964844 nb_pixel_total : 63459 time to create 1 rle with old method : 0.07275629043579102 create new chi : 3.1085658073425293 time to delete rle : 0.0014734268188476562 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 9501 TO DO : save crop sub photo not yet done ! save time : 0.559786319732666 nb_obj : 22 nb_hashtags : 6 time to prepare the origin masks : 10.466492176055908 time for calcul the mask position with numpy : 0.617272138595581 nb_pixel_total : 5886296 time to create 1 rle with new method : 0.8965277671813965 time for calcul the mask position with numpy : 0.027788877487182617 nb_pixel_total : 39913 time to create 1 rle with old method : 0.04254269599914551 time for calcul the mask position with numpy : 0.026239395141601562 nb_pixel_total : 11 time to create 1 rle with old method : 9.655952453613281e-05 time for calcul the mask position with numpy : 0.02813410758972168 nb_pixel_total : 22842 time to create 1 rle with old method : 0.02636432647705078 time for calcul the mask position with numpy : 0.028392314910888672 nb_pixel_total : 1512 time to create 1 rle with old method : 0.0021457672119140625 time for calcul the mask position with numpy : 0.046144723892211914 nb_pixel_total : 20270 time to create 1 rle with old method : 0.022780418395996094 time for calcul the mask position with numpy : 0.04716682434082031 nb_pixel_total : 162706 time to create 1 rle with new method : 0.715477466583252 time for calcul the mask position with numpy : 0.03572392463684082 nb_pixel_total : 53630 time to create 1 rle with old method : 0.0567166805267334 time for calcul the mask position with numpy : 0.03637838363647461 nb_pixel_total : 134426 time to create 1 rle with old method : 0.1427004337310791 time for calcul the mask position with numpy : 0.03695797920227051 nb_pixel_total : 18242 time to create 1 rle with old method : 0.020279645919799805 time for calcul the mask position with numpy : 0.03753256797790527 nb_pixel_total : 104132 time to create 1 rle with old method : 0.10954642295837402 time for calcul the mask position with numpy : 0.035059213638305664 nb_pixel_total : 19918 time to create 1 rle with old method : 0.02204728126525879 time for calcul the mask position with numpy : 0.040975332260131836 nb_pixel_total : 185987 time to create 1 rle with new method : 0.44962239265441895 time for calcul the mask position with numpy : 0.027374744415283203 nb_pixel_total : 39349 time to create 1 rle with old method : 0.04244494438171387 time for calcul the mask position with numpy : 0.022294044494628906 nb_pixel_total : 128845 time to create 1 rle with old method : 0.13512611389160156 time for calcul the mask position with numpy : 0.021869182586669922 nb_pixel_total : 2265 time to create 1 rle with old method : 0.0024275779724121094 time for calcul the mask position with numpy : 0.025397777557373047 nb_pixel_total : 60190 time to create 1 rle with old method : 0.06337261199951172 time for calcul the mask position with numpy : 0.025507450103759766 nb_pixel_total : 1597 time to create 1 rle with old method : 0.001873016357421875 time for calcul the mask position with numpy : 0.027716636657714844 nb_pixel_total : 32760 time to create 1 rle with old method : 0.036145687103271484 time for calcul the mask position with numpy : 0.025743961334228516 nb_pixel_total : 59926 time to create 1 rle with old method : 0.06377601623535156 time for calcul the mask position with numpy : 0.027068614959716797 nb_pixel_total : 5668 time to create 1 rle with old method : 0.006471395492553711 time for calcul the mask position with numpy : 0.030716419219970703 nb_pixel_total : 5507 time to create 1 rle with old method : 0.006409645080566406 time for calcul the mask position with numpy : 0.03731966018676758 nb_pixel_total : 64248 time to create 1 rle with old method : 0.0687711238861084 create new chi : 4.350318670272827 time to delete rle : 0.0014247894287109375 batch 1 Loaded 23 chid ids of type : 3726 Number RLEs to save : 14674 TO DO : save crop sub photo not yet done ! save time : 1.073225975036621 map_output_result : {1335314346: (0.0, 'Should be the crop_list due to order', 0.0), 1335314324: (0.0, 'Should be the crop_list due to order', 0.0), 1335314322: (0.0, 'Should be the crop_list due to order', 0.0), 1335314302: (0.0, 'Should be the crop_list due to order', 0.0), 1335314295: (0.0, 'Should be the crop_list due to order', 0.0), 1335314287: (0.0, 'Should be the crop_list due to order', 0.0), 1335314283: (0.0, 'Should be the crop_list due to order', 0.0), 1335314281: (0.0, 'Should be the crop_list due to order', 0.0), 1335314278: (0.0, 'Should be the crop_list due to order', 0.0), 1335314182: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1335314346, 1335314324, 1335314322, 1335314302, 1335314295, 1335314287, 1335314283, 1335314281, 1335314278, 1335314182] Looping around the photos to save general results len do output : 10 /1335314346.Didn't retrieve data . /1335314324.Didn't retrieve data . /1335314322.Didn't retrieve data . /1335314302.Didn't retrieve data . /1335314295.Didn't retrieve data . /1335314287.Didn't retrieve data . /1335314283.Didn't retrieve data . /1335314281.Didn't retrieve data . /1335314278.Didn't retrieve data . /1335314182.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314346', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314324', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314322', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314302', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314295', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314287', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314283', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314281', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314278', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314182', None, None, None, None, None, '2560904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.0833890438079834 save_final save missing photos in datou_result : time spend for datou_step_exec : 140.57632875442505 time spend to save output : 0.26450419425964355 total time spend for step 5 : 140.8408329486847 step6:crop_condition Fri Feb 7 01:37:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 183 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888627_923107 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.8483545780181885 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888636_923107 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.9113492965698242 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888641_923107 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.710040807723999 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888646_923107 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 3.13771653175354 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 4869462 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888651_923107 we have uploaded 9 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.354947328567505 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888656_923107 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.1160004138946533 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888659_923107 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.0066232681274414 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335314346, 1335314324, 1335314322, 1335314302, 1335314295, 1335314287, 1335314283, 1335314281, 1335314278, 1335314182] Looping around the photos to save general results len do output : 38 /1335331882Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331883Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331884Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331885Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331886Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331892Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331893Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331894Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331895Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331896Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331897Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331898Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331901Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331902Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331904Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331905Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331906Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331911Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331912Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331913Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331914Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331915Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331916Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331917Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331921Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331922Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331923Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331924Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331925Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331926Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331927Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331928Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331929Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331953Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331954Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331955Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331958Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335331959Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314346', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314324', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314322', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314302', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314295', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314287', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314283', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314281', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314278', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314182', None, None, None, None, None, '2560904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 124 time used for this insertion : 0.03442835807800293 save_final save missing photos in datou_result : time spend for datou_step_exec : 40.018041133880615 time spend to save output : 0.20993375778198242 total time spend for step 6 : 40.2279748916626 step7:ventilate_hashtags_in_portfolio Fri Feb 7 01:37:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20314846 get user id for portfolio 20314846 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314846 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','autre_refus','kraft','cartonnette','mal_croppe','environnement','flou','plastique','Teint_Dans_La_Masse','metal','Carton_brun','Carton_gris')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314846 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','autre_refus','kraft','cartonnette','mal_croppe','environnement','flou','plastique','Teint_Dans_La_Masse','metal','Carton_brun','Carton_gris')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314846 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','autre_refus','kraft','cartonnette','mal_croppe','environnement','flou','plastique','Teint_Dans_La_Masse','metal','Carton_brun','Carton_gris')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20314897,20314898,20314899,20314900,20314901,20314902,20314903,20314904,20314905,20314906,20314907,20314908?tags=papier,autre_refus,kraft,cartonnette,mal_croppe,environnement,flou,plastique,Teint_Dans_La_Masse,metal,Carton_brun,Carton_gris Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1335314346, 1335314324, 1335314322, 1335314302, 1335314295, 1335314287, 1335314283, 1335314281, 1335314278, 1335314182] Looping around the photos to save general results len do output : 1 /20314846. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314346', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314324', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314322', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314302', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314295', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314287', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314283', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314281', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314278', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314182', None, None, None, None, None, '2560904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014552593231201172 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.9125449657440186 time spend to save output : 0.01490163803100586 total time spend for step 7 : 0.9274466037750244 step8:final Fri Feb 7 01:37:45 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1335314346: ('0.09628085696972587',), 1335314324: ('0.09628085696972587',), 1335314322: ('0.09628085696972587',), 1335314302: ('0.09628085696972587',), 1335314295: ('0.09628085696972587',), 1335314287: ('0.09628085696972587',), 1335314283: ('0.09628085696972587',), 1335314281: ('0.09628085696972587',), 1335314278: ('0.09628085696972587',), 1335314182: ('0.09628085696972587',)} new output for save of step final : {1335314346: ('0.09628085696972587',), 1335314324: ('0.09628085696972587',), 1335314322: ('0.09628085696972587',), 1335314302: ('0.09628085696972587',), 1335314295: ('0.09628085696972587',), 1335314287: ('0.09628085696972587',), 1335314283: ('0.09628085696972587',), 1335314281: ('0.09628085696972587',), 1335314278: ('0.09628085696972587',), 1335314182: ('0.09628085696972587',)} [1335314346, 1335314324, 1335314322, 1335314302, 1335314295, 1335314287, 1335314283, 1335314281, 1335314278, 1335314182] Looping around the photos to save general results len do output : 10 /1335314346.Didn't retrieve data . /1335314324.Didn't retrieve data . /1335314322.Didn't retrieve data . /1335314302.Didn't retrieve data . /1335314295.Didn't retrieve data . /1335314287.Didn't retrieve data . /1335314283.Didn't retrieve data . /1335314281.Didn't retrieve data . /1335314278.Didn't retrieve data . /1335314182.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314346', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314324', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314322', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314302', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314295', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314287', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314283', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314281', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314278', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314182', None, None, None, None, None, '2560904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013278722763061523 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11577582359313965 time spend to save output : 0.013820171356201172 total time spend for step 8 : 0.12959599494934082 step9:velours_tree Fri Feb 7 01:37:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.18485116958618164 time spend to save output : 3.695487976074219e-05 total time spend for step 9 : 0.18488812446594238 step10:send_mail_cod Fri Feb 7 01:37:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20314846_07-02-2025_01_37_46.pdf 20314898 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203148981738888666 20314899 change filename to text .change filename to text .imagette203148991738888666 20314900 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203149001738888666 20314901 imagette203149011738888667 20314903 imagette203149031738888667 20314904 change filename to text .change filename to text .change filename to text .imagette203149041738888667 20314906 imagette203149061738888667 20314905 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203149051738888667 20314907 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203149071738888667 20314908 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203149081738888668 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20314846 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20314897,20314898,20314899,20314900,20314901,20314902,20314903,20314904,20314905,20314906,20314907,20314908?tags=papier,autre_refus,kraft,cartonnette,mal_croppe,environnement,flou,plastique,Teint_Dans_La_Masse,metal,Carton_brun,Carton_gris your option no_mail is active, we will not send the real mail to your client args[1335314346] : ('0.09628085696972587',) no score found for photo 1335314346 apple ('0.09628085696972587',) We are sending mail with results at report@fotonower.com args[1335314324] : ('0.09628085696972587',) no score found for photo 1335314324 apple ('0.09628085696972587',) We are sending mail with results at report@fotonower.com args[1335314322] : ('0.09628085696972587',) no score found for photo 1335314322 apple ('0.09628085696972587',) We are sending mail with results at report@fotonower.com args[1335314302] : ('0.09628085696972587',) no score found for photo 1335314302 apple ('0.09628085696972587',) We are sending mail with results at report@fotonower.com args[1335314295] : ('0.09628085696972587',) no score found for photo 1335314295 apple ('0.09628085696972587',) We are sending mail with results at report@fotonower.com args[1335314287] : ('0.09628085696972587',) no score found for photo 1335314287 apple ('0.09628085696972587',) We are sending mail with results at report@fotonower.com args[1335314283] : ('0.09628085696972587',) no score found for photo 1335314283 apple ('0.09628085696972587',) We are sending mail with results at report@fotonower.com args[1335314281] : ('0.09628085696972587',) no score found for photo 1335314281 apple ('0.09628085696972587',) We are sending mail with results at report@fotonower.com args[1335314278] : ('0.09628085696972587',) no score found for photo 1335314278 apple ('0.09628085696972587',) We are sending mail with results at report@fotonower.com args[1335314182] : ('0.09628085696972587',) no score found for photo 1335314182 apple ('0.09628085696972587',) We are sending mail with results at report@fotonower.com refus_total : 0.09628085696972587 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20314846 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1335237948,1335314346,1335314295,1335314283,1335314281,1335314278,1335314182,1335314166,1335237968,1335237684,1335237670,1335237475,1335237271,1335237469,1335237598,1335237610,1335237257,1335237874,1335314147,1335314164) Found this number of photos: 20 begin to download photo : 1335237948 begin to download photo : 1335314278 begin to download photo : 1335237670 begin to download photo : 1335237610 download finish for photo 1335237670 begin to download photo : 1335237475 download finish for photo 1335314278 begin to download photo : 1335314182 download finish for photo 1335237948 begin to download photo : 1335314346 download finish for photo 1335237610 begin to download photo : 1335237257 download finish for photo 1335314182 begin to download photo : 1335314166 download finish for photo 1335237475 begin to download photo : 1335237271 download finish for photo 1335237257 begin to download photo : 1335237874 download finish for photo 1335314346 begin to download photo : 1335314295 download finish for photo 1335314166 begin to download photo : 1335237968 download finish for photo 1335237271 begin to download photo : 1335237469 download finish for photo 1335314295 begin to download photo : 1335314283 download finish for photo 1335237874 begin to download photo : 1335314147 download finish for photo 1335237968 begin to download photo : 1335237684 download finish for photo 1335237469 begin to download photo : 1335237598 download finish for photo 1335314283 begin to download photo : 1335314281 download finish for photo 1335314147 begin to download photo : 1335314164 download finish for photo 1335237684 download finish for photo 1335237598 download finish for photo 1335314281 download finish for photo 1335314164 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314846_07-02-2025_01_37_46.pdf results_Auto_P20314846_07-02-2025_01_37_46.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314846_07-02-2025_01_37_46.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20314846','results_Auto_P20314846_07-02-2025_01_37_46.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314846_07-02-2025_01_37_46.pdf','pdf','','0.77','0.09628085696972587') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1335314346, 1335314324, 1335314322, 1335314302, 1335314295, 1335314287, 1335314283, 1335314281, 1335314278, 1335314182] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314346', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314324', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314322', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314302', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314295', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314287', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314283', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314281', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314278', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314182', None, None, None, None, None, '2560904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.1258087158203125 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.334454774856567 time spend to save output : 0.1262044906616211 total time spend for step 10 : 6.4606592655181885 step11:split_time_score Fri Feb 7 01:37:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('17', 30),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 06022025 20314846 Nombre de photos uploadées : 30 / 23040 (0%) 06022025 20314846 Nombre de photos taguées (types de déchets): 0 / 30 (0%) 06022025 20314846 Nombre de photos taguées (volume) : 0 / 30 (0%) elapsed_time : load_data_split_time_score 2.384185791015625e-06 elapsed_time : order_list_meta_photo_and_scores 1.2874603271484375e-05 ?????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0017001628875732422 elapsed_time : insert_dashboard_record_day_entry 0.023037433624267578 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314812 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314813 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314814 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314815 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314816 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314817 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314818 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314819 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314820 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314821 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314822 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314823 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314824 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314825 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314826 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314827 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314828 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314829 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314830 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314831 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314832 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314833 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314834 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314835 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314836 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314837 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314838 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314839 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314840 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314841 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314842 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314843 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314844 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314845 order by id desc limit 1 Qualite : 0.09628085696972587 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314846_07-02-2025_01_37_46.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314846 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314846 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314847 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314848 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314849 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314850 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314851 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314852 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314853 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314854 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314855 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314856 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314857 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'06022025': {'nb_upload': 30, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1335314346, 1335314324, 1335314322, 1335314302, 1335314295, 1335314287, 1335314283, 1335314281, 1335314278, 1335314182] Looping around the photos to save general results len do output : 1 /20314846Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314346', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314324', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314322', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314302', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314295', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314287', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314283', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314281', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314278', None, None, None, None, None, '2560904') ('3459', None, None, None, None, None, None, None, '2560904') ('3459', '20314846', '1335314182', None, None, None, None, None, '2560904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013412714004516602 save_final save missing photos in datou_result : time spend for datou_step_exec : 36.21336007118225 time spend to save output : 0.01378774642944336 total time spend for step 11 : 36.227147817611694 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 178.13user 116.60system 6:05.54elapsed 80%CPU (0avgtext+0avgdata 4276392maxresident)k 3084664inputs+135824outputs (24019major+13138442minor)pagefaults 0swaps