python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 32503 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 20>10 please execute split_portfolio.py -i 20461697 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2577749'] with mtr_portfolio_ids : ['20461697'] and first list_photo_ids : [] new path : /proc/32503/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.429636240005493 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue Feb 11 21:16:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4822 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-11 21:16:37.546954: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-11 21:16:37.779211: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-11 21:16:37.781931: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f1ecc000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-11 21:16:37.781993: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-11 21:16:37.797594: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-11 21:16:38.239991: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x350ab630 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-11 21:16:38.240058: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-11 21:16:38.242017: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 21:16:38.245048: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 21:16:38.305670: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 21:16:38.335694: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 21:16:38.341694: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 21:16:38.382949: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 21:16:38.389082: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 21:16:38.456927: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 21:16:38.458183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 21:16:38.458599: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 21:16:38.460064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-11 21:16:38.460083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-11 21:16:38.460092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-11 21:16:38.461971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4370 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-11 21:16:38.987109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 21:16:38.987207: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 21:16:38.987230: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 21:16:38.987248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 21:16:38.987266: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 21:16:38.987284: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 21:16:38.987301: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 21:16:38.987319: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 21:16:38.988458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 21:16:38.989726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 21:16:38.989800: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 21:16:38.989823: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 21:16:38.989844: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 21:16:38.989863: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 21:16:38.989883: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 21:16:38.989903: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 21:16:38.989922: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 21:16:38.991058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 21:16:38.991093: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-11 21:16:38.991104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-11 21:16:38.991114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-11 21:16:38.992329: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4370 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-11 21:16:44.595056: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 51380224 exceeds 10% of free system memory. 2025-02-11 21:16:51.385964: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 21:16:51.806077: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 21:16:54.185209: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.77G (4047372288 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 27 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 24 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 21 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 21 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 11 Detection mask done ! Trying to reset tf kernel 32835 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 155 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4822 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.00047779083251953125 nb_pixel_total : 6477 time to create 1 rle with old method : 0.010127782821655273 length of segment : 99 time for calcul the mask position with numpy : 0.0019180774688720703 nb_pixel_total : 68734 time to create 1 rle with old method : 0.0868067741394043 length of segment : 342 time for calcul the mask position with numpy : 0.017389535903930664 nb_pixel_total : 298544 time to create 1 rle with new method : 0.032694339752197266 length of segment : 626 time for calcul the mask position with numpy : 0.00019073486328125 nb_pixel_total : 6414 time to create 1 rle with old method : 0.008429765701293945 length of segment : 85 time for calcul the mask position with numpy : 0.0018150806427001953 nb_pixel_total : 109141 time to create 1 rle with old method : 0.1412205696105957 length of segment : 436 time for calcul the mask position with numpy : 0.006101369857788086 nb_pixel_total : 316901 time to create 1 rle with new method : 0.04047727584838867 length of segment : 688 time for calcul the mask position with numpy : 0.0011365413665771484 nb_pixel_total : 37815 time to create 1 rle with old method : 0.04769587516784668 length of segment : 455 time for calcul the mask position with numpy : 0.0027120113372802734 nb_pixel_total : 151336 time to create 1 rle with new method : 0.008777618408203125 length of segment : 613 time for calcul the mask position with numpy : 0.00045371055603027344 nb_pixel_total : 16952 time to create 1 rle with old method : 0.025496244430541992 length of segment : 170 time for calcul the mask position with numpy : 0.0006892681121826172 nb_pixel_total : 39302 time to create 1 rle with old method : 0.04592132568359375 length of segment : 354 time for calcul the mask position with numpy : 0.00030994415283203125 nb_pixel_total : 11671 time to create 1 rle with old method : 0.014638423919677734 length of segment : 103 time for calcul the mask position with numpy : 0.0002522468566894531 nb_pixel_total : 5347 time to create 1 rle with old method : 0.0072667598724365234 length of segment : 181 time for calcul the mask position with numpy : 0.00011301040649414062 nb_pixel_total : 2762 time to create 1 rle with old method : 0.0036649703979492188 length of segment : 63 time for calcul the mask position with numpy : 0.0017499923706054688 nb_pixel_total : 94808 time to create 1 rle with old method : 0.10840773582458496 length of segment : 609 time for calcul the mask position with numpy : 0.0004055500030517578 nb_pixel_total : 21723 time to create 1 rle with old method : 0.0267486572265625 length of segment : 227 time for calcul the mask position with numpy : 0.0003085136413574219 nb_pixel_total : 7244 time to create 1 rle with old method : 0.00940084457397461 length of segment : 115 time for calcul the mask position with numpy : 0.0007405281066894531 nb_pixel_total : 36526 time to create 1 rle with old method : 0.041941165924072266 length of segment : 337 time for calcul the mask position with numpy : 0.0003695487976074219 nb_pixel_total : 16456 time to create 1 rle with old method : 0.01924443244934082 length of segment : 143 time for calcul the mask position with numpy : 0.0009224414825439453 nb_pixel_total : 58235 time to create 1 rle with old method : 0.07184028625488281 length of segment : 286 time for calcul the mask position with numpy : 0.0006496906280517578 nb_pixel_total : 21530 time to create 1 rle with old method : 0.027462244033813477 length of segment : 169 time for calcul the mask position with numpy : 0.0001773834228515625 nb_pixel_total : 4280 time to create 1 rle with old method : 0.010221004486083984 length of segment : 58 time for calcul the mask position with numpy : 0.0006961822509765625 nb_pixel_total : 10687 time to create 1 rle with old method : 0.012957334518432617 length of segment : 99 time for calcul the mask position with numpy : 0.0007948875427246094 nb_pixel_total : 10629 time to create 1 rle with old method : 0.013491153717041016 length of segment : 135 time for calcul the mask position with numpy : 0.002907276153564453 nb_pixel_total : 63180 time to create 1 rle with old method : 0.07623648643493652 length of segment : 443 time for calcul the mask position with numpy : 0.0004601478576660156 nb_pixel_total : 9717 time to create 1 rle with old method : 0.011812448501586914 length of segment : 102 time for calcul the mask position with numpy : 0.0005621910095214844 nb_pixel_total : 4975 time to create 1 rle with old method : 0.006946086883544922 length of segment : 148 time for calcul the mask position with numpy : 0.0003116130828857422 nb_pixel_total : 2795 time to create 1 rle with old method : 0.005282402038574219 length of segment : 70 time for calcul the mask position with numpy : 0.005190610885620117 nb_pixel_total : 190228 time to create 1 rle with new method : 0.011651039123535156 length of segment : 442 time for calcul the mask position with numpy : 0.0019178390502929688 nb_pixel_total : 52781 time to create 1 rle with old method : 0.06178426742553711 length of segment : 358 time for calcul the mask position with numpy : 0.0016531944274902344 nb_pixel_total : 56155 time to create 1 rle with old method : 0.0661931037902832 length of segment : 268 time for calcul the mask position with numpy : 0.0019023418426513672 nb_pixel_total : 34589 time to create 1 rle with old method : 0.040024518966674805 length of segment : 202 time for calcul the mask position with numpy : 0.0005133152008056641 nb_pixel_total : 9706 time to create 1 rle with old method : 0.011614322662353516 length of segment : 110 time for calcul the mask position with numpy : 0.0030715465545654297 nb_pixel_total : 69514 time to create 1 rle with old method : 0.07921504974365234 length of segment : 440 time for calcul the mask position with numpy : 0.00018787384033203125 nb_pixel_total : 3751 time to create 1 rle with old method : 0.0045812129974365234 length of segment : 62 time for calcul the mask position with numpy : 0.001489877700805664 nb_pixel_total : 32973 time to create 1 rle with old method : 0.041449785232543945 length of segment : 305 time for calcul the mask position with numpy : 0.0006392002105712891 nb_pixel_total : 10889 time to create 1 rle with old method : 0.018285512924194336 length of segment : 140 time for calcul the mask position with numpy : 0.0016171932220458984 nb_pixel_total : 25139 time to create 1 rle with old method : 0.02985548973083496 length of segment : 291 time for calcul the mask position with numpy : 0.00018095970153808594 nb_pixel_total : 2536 time to create 1 rle with old method : 0.0032482147216796875 length of segment : 57 time for calcul the mask position with numpy : 0.0012028217315673828 nb_pixel_total : 43077 time to create 1 rle with old method : 0.050200700759887695 length of segment : 421 time for calcul the mask position with numpy : 0.00033926963806152344 nb_pixel_total : 6171 time to create 1 rle with old method : 0.0077741146087646484 length of segment : 74 time for calcul the mask position with numpy : 0.14220571517944336 nb_pixel_total : 352961 time to create 1 rle with new method : 0.029817819595336914 length of segment : 658 time for calcul the mask position with numpy : 0.0026345252990722656 nb_pixel_total : 77390 time to create 1 rle with old method : 0.0876622200012207 length of segment : 518 time for calcul the mask position with numpy : 0.0004839897155761719 nb_pixel_total : 18629 time to create 1 rle with old method : 0.02157759666442871 length of segment : 311 time for calcul the mask position with numpy : 0.0038695335388183594 nb_pixel_total : 154307 time to create 1 rle with new method : 0.008232355117797852 length of segment : 646 time for calcul the mask position with numpy : 0.0014405250549316406 nb_pixel_total : 45179 time to create 1 rle with old method : 0.05250358581542969 length of segment : 291 time for calcul the mask position with numpy : 0.08823442459106445 nb_pixel_total : 1915990 time to create 1 rle with new method : 0.1889357566833496 length of segment : 2618 time for calcul the mask position with numpy : 0.0005331039428710938 nb_pixel_total : 27203 time to create 1 rle with old method : 0.03142857551574707 length of segment : 145 time for calcul the mask position with numpy : 0.0003943443298339844 nb_pixel_total : 19921 time to create 1 rle with old method : 0.02378225326538086 length of segment : 158 time for calcul the mask position with numpy : 0.0004899501800537109 nb_pixel_total : 21343 time to create 1 rle with old method : 0.02532339096069336 length of segment : 157 time for calcul the mask position with numpy : 0.0014562606811523438 nb_pixel_total : 70505 time to create 1 rle with old method : 0.07945466041564941 length of segment : 475 time for calcul the mask position with numpy : 0.00039577484130859375 nb_pixel_total : 20646 time to create 1 rle with old method : 0.030702590942382812 length of segment : 162 time for calcul the mask position with numpy : 0.00016546249389648438 nb_pixel_total : 1848 time to create 1 rle with old method : 0.0024881362915039062 length of segment : 72 time for calcul the mask position with numpy : 0.008605480194091797 nb_pixel_total : 370591 time to create 1 rle with new method : 0.017791748046875 length of segment : 506 time for calcul the mask position with numpy : 0.00054931640625 nb_pixel_total : 29806 time to create 1 rle with old method : 0.03468823432922363 length of segment : 224 time for calcul the mask position with numpy : 0.0011484622955322266 nb_pixel_total : 72451 time to create 1 rle with old method : 0.08194279670715332 length of segment : 465 time for calcul the mask position with numpy : 0.0018079280853271484 nb_pixel_total : 46788 time to create 1 rle with old method : 0.059592485427856445 length of segment : 309 time for calcul the mask position with numpy : 0.4001431465148926 nb_pixel_total : 3835515 time to create 1 rle with new method : 0.4394702911376953 length of segment : 3417 time for calcul the mask position with numpy : 0.001973390579223633 nb_pixel_total : 75817 time to create 1 rle with old method : 0.11339950561523438 length of segment : 653 time for calcul the mask position with numpy : 0.00019884109497070312 nb_pixel_total : 2586 time to create 1 rle with old method : 0.0033190250396728516 length of segment : 64 time for calcul the mask position with numpy : 0.001613616943359375 nb_pixel_total : 100642 time to create 1 rle with old method : 0.1149282455444336 length of segment : 374 time for calcul the mask position with numpy : 0.00045800209045410156 nb_pixel_total : 15470 time to create 1 rle with old method : 0.018424510955810547 length of segment : 173 time for calcul the mask position with numpy : 0.0017750263214111328 nb_pixel_total : 61672 time to create 1 rle with old method : 0.07686257362365723 length of segment : 300 time for calcul the mask position with numpy : 0.000989675521850586 nb_pixel_total : 32351 time to create 1 rle with old method : 0.0375213623046875 length of segment : 238 time for calcul the mask position with numpy : 0.0003180503845214844 nb_pixel_total : 8054 time to create 1 rle with old method : 0.010504484176635742 length of segment : 61 time for calcul the mask position with numpy : 0.002157449722290039 nb_pixel_total : 69641 time to create 1 rle with old method : 0.10012674331665039 length of segment : 370 time for calcul the mask position with numpy : 0.00020074844360351562 nb_pixel_total : 2815 time to create 1 rle with old method : 0.0037567615509033203 length of segment : 89 time for calcul the mask position with numpy : 0.0002899169921875 nb_pixel_total : 6284 time to create 1 rle with old method : 0.007990121841430664 length of segment : 98 time for calcul the mask position with numpy : 0.0002276897430419922 nb_pixel_total : 2273 time to create 1 rle with old method : 0.0027735233306884766 length of segment : 110 time for calcul the mask position with numpy : 0.00313568115234375 nb_pixel_total : 86733 time to create 1 rle with old method : 0.10488343238830566 length of segment : 451 time for calcul the mask position with numpy : 0.0006718635559082031 nb_pixel_total : 18166 time to create 1 rle with old method : 0.02137899398803711 length of segment : 164 time for calcul the mask position with numpy : 0.021651506423950195 nb_pixel_total : 195219 time to create 1 rle with new method : 0.8878538608551025 length of segment : 792 time for calcul the mask position with numpy : 0.0008325576782226562 nb_pixel_total : 38450 time to create 1 rle with old method : 0.044191598892211914 length of segment : 229 time for calcul the mask position with numpy : 0.0005464553833007812 nb_pixel_total : 38727 time to create 1 rle with old method : 0.04480481147766113 length of segment : 225 time for calcul the mask position with numpy : 0.002332448959350586 nb_pixel_total : 112033 time to create 1 rle with old method : 0.13129806518554688 length of segment : 288 time for calcul the mask position with numpy : 0.0005133152008056641 nb_pixel_total : 9088 time to create 1 rle with old method : 0.011654138565063477 length of segment : 221 time for calcul the mask position with numpy : 0.007325172424316406 nb_pixel_total : 245951 time to create 1 rle with new method : 0.0534520149230957 length of segment : 799 time for calcul the mask position with numpy : 0.00026702880859375 nb_pixel_total : 8219 time to create 1 rle with old method : 0.009752273559570312 length of segment : 124 time for calcul the mask position with numpy : 0.0005671977996826172 nb_pixel_total : 12714 time to create 1 rle with old method : 0.015095233917236328 length of segment : 146 time for calcul the mask position with numpy : 0.0005724430084228516 nb_pixel_total : 12644 time to create 1 rle with old method : 0.01578974723815918 length of segment : 167 time for calcul the mask position with numpy : 0.0007109642028808594 nb_pixel_total : 12414 time to create 1 rle with old method : 0.015991926193237305 length of segment : 135 time for calcul the mask position with numpy : 0.0013568401336669922 nb_pixel_total : 38941 time to create 1 rle with old method : 0.04566359519958496 length of segment : 199 time for calcul the mask position with numpy : 0.0005686283111572266 nb_pixel_total : 15424 time to create 1 rle with old method : 0.019164085388183594 length of segment : 151 time for calcul the mask position with numpy : 0.005232334136962891 nb_pixel_total : 189308 time to create 1 rle with new method : 0.01081085205078125 length of segment : 673 time for calcul the mask position with numpy : 0.002193450927734375 nb_pixel_total : 45760 time to create 1 rle with old method : 0.05559372901916504 length of segment : 318 time for calcul the mask position with numpy : 0.00012350082397460938 nb_pixel_total : 3259 time to create 1 rle with old method : 0.004257917404174805 length of segment : 60 time for calcul the mask position with numpy : 0.003082275390625 nb_pixel_total : 102282 time to create 1 rle with old method : 0.11722946166992188 length of segment : 353 time for calcul the mask position with numpy : 0.0002818107604980469 nb_pixel_total : 5976 time to create 1 rle with old method : 0.007601022720336914 length of segment : 155 time for calcul the mask position with numpy : 0.0004928112030029297 nb_pixel_total : 7283 time to create 1 rle with old method : 0.009006738662719727 length of segment : 118 time for calcul the mask position with numpy : 0.0002510547637939453 nb_pixel_total : 3412 time to create 1 rle with old method : 0.004461526870727539 length of segment : 87 time for calcul the mask position with numpy : 0.0016529560089111328 nb_pixel_total : 53353 time to create 1 rle with old method : 0.06640195846557617 length of segment : 278 time for calcul the mask position with numpy : 0.0018265247344970703 nb_pixel_total : 51019 time to create 1 rle with old method : 0.05998730659484863 length of segment : 696 time for calcul the mask position with numpy : 0.00046181678771972656 nb_pixel_total : 1263 time to create 1 rle with old method : 0.0019447803497314453 length of segment : 106 time for calcul the mask position with numpy : 0.0033447742462158203 nb_pixel_total : 98485 time to create 1 rle with old method : 0.1143045425415039 length of segment : 344 time for calcul the mask position with numpy : 0.002142190933227539 nb_pixel_total : 64259 time to create 1 rle with old method : 0.07465481758117676 length of segment : 362 time for calcul the mask position with numpy : 0.0004572868347167969 nb_pixel_total : 9271 time to create 1 rle with old method : 0.011004447937011719 length of segment : 119 time for calcul the mask position with numpy : 0.0009648799896240234 nb_pixel_total : 21216 time to create 1 rle with old method : 0.024414539337158203 length of segment : 243 time for calcul the mask position with numpy : 0.0010242462158203125 nb_pixel_total : 30311 time to create 1 rle with old method : 0.036354780197143555 length of segment : 210 time for calcul the mask position with numpy : 0.00017404556274414062 nb_pixel_total : 1656 time to create 1 rle with old method : 0.002249002456665039 length of segment : 56 time for calcul the mask position with numpy : 0.04640603065490723 nb_pixel_total : 1502459 time to create 1 rle with new method : 0.1318516731262207 length of segment : 2368 time for calcul the mask position with numpy : 0.0036301612854003906 nb_pixel_total : 152820 time to create 1 rle with new method : 0.005549430847167969 length of segment : 407 time for calcul the mask position with numpy : 0.0003285408020019531 nb_pixel_total : 6220 time to create 1 rle with old method : 0.007616758346557617 length of segment : 61 time for calcul the mask position with numpy : 0.0015289783477783203 nb_pixel_total : 53857 time to create 1 rle with old method : 0.06170654296875 length of segment : 333 time for calcul the mask position with numpy : 0.00047516822814941406 nb_pixel_total : 9418 time to create 1 rle with old method : 0.012107610702514648 length of segment : 130 time for calcul the mask position with numpy : 0.0001308917999267578 nb_pixel_total : 2230 time to create 1 rle with old method : 0.00298309326171875 length of segment : 65 time for calcul the mask position with numpy : 0.002530813217163086 nb_pixel_total : 118446 time to create 1 rle with old method : 0.15640997886657715 length of segment : 626 time for calcul the mask position with numpy : 0.0002753734588623047 nb_pixel_total : 4702 time to create 1 rle with old method : 0.0062084197998046875 length of segment : 46 time for calcul the mask position with numpy : 0.0023658275604248047 nb_pixel_total : 82912 time to create 1 rle with old method : 0.10125064849853516 length of segment : 479 time for calcul the mask position with numpy : 0.0002720355987548828 nb_pixel_total : 3883 time to create 1 rle with old method : 0.007010221481323242 length of segment : 115 time for calcul the mask position with numpy : 0.0017371177673339844 nb_pixel_total : 51291 time to create 1 rle with old method : 0.06350922584533691 length of segment : 251 time for calcul the mask position with numpy : 0.0021071434020996094 nb_pixel_total : 95631 time to create 1 rle with old method : 0.13630104064941406 length of segment : 429 time for calcul the mask position with numpy : 0.0027191638946533203 nb_pixel_total : 121215 time to create 1 rle with old method : 0.14680790901184082 length of segment : 646 time for calcul the mask position with numpy : 0.0031561851501464844 nb_pixel_total : 123409 time to create 1 rle with old method : 0.17076539993286133 length of segment : 721 time for calcul the mask position with numpy : 0.0040018558502197266 nb_pixel_total : 114972 time to create 1 rle with old method : 0.14585661888122559 length of segment : 383 time for calcul the mask position with numpy : 0.0012726783752441406 nb_pixel_total : 57898 time to create 1 rle with old method : 0.07271051406860352 length of segment : 501 time for calcul the mask position with numpy : 0.0003266334533691406 nb_pixel_total : 3058 time to create 1 rle with old method : 0.004270315170288086 length of segment : 101 time for calcul the mask position with numpy : 0.015334129333496094 nb_pixel_total : 538226 time to create 1 rle with new method : 0.0333249568939209 length of segment : 1566 time for calcul the mask position with numpy : 0.002233743667602539 nb_pixel_total : 54233 time to create 1 rle with old method : 0.06352424621582031 length of segment : 366 time for calcul the mask position with numpy : 0.002347707748413086 nb_pixel_total : 56545 time to create 1 rle with old method : 0.06860518455505371 length of segment : 427 time for calcul the mask position with numpy : 0.003031492233276367 nb_pixel_total : 71673 time to create 1 rle with old method : 0.09353494644165039 length of segment : 380 time for calcul the mask position with numpy : 0.0008118152618408203 nb_pixel_total : 20294 time to create 1 rle with old method : 0.02603936195373535 length of segment : 184 time for calcul the mask position with numpy : 0.00019431114196777344 nb_pixel_total : 2879 time to create 1 rle with old method : 0.004809141159057617 length of segment : 88 time for calcul the mask position with numpy : 0.001712799072265625 nb_pixel_total : 51372 time to create 1 rle with old method : 0.06589937210083008 length of segment : 421 time for calcul the mask position with numpy : 0.003729581832885742 nb_pixel_total : 137538 time to create 1 rle with old method : 0.18485760688781738 length of segment : 580 time for calcul the mask position with numpy : 0.0015499591827392578 nb_pixel_total : 46400 time to create 1 rle with old method : 0.05518364906311035 length of segment : 315 time for calcul the mask position with numpy : 0.0010406970977783203 nb_pixel_total : 22320 time to create 1 rle with old method : 0.02677607536315918 length of segment : 330 time for calcul the mask position with numpy : 0.0003159046173095703 nb_pixel_total : 9546 time to create 1 rle with old method : 0.011910438537597656 length of segment : 104 time for calcul the mask position with numpy : 0.0007746219635009766 nb_pixel_total : 21058 time to create 1 rle with old method : 0.024376392364501953 length of segment : 220 time for calcul the mask position with numpy : 0.0006968975067138672 nb_pixel_total : 25632 time to create 1 rle with old method : 0.029712915420532227 length of segment : 284 time for calcul the mask position with numpy : 0.0002899169921875 nb_pixel_total : 17018 time to create 1 rle with old method : 0.02197885513305664 length of segment : 173 time for calcul the mask position with numpy : 0.00036644935607910156 nb_pixel_total : 8484 time to create 1 rle with old method : 0.014794588088989258 length of segment : 136 time for calcul the mask position with numpy : 0.0006983280181884766 nb_pixel_total : 11341 time to create 1 rle with old method : 0.017060041427612305 length of segment : 149 time for calcul the mask position with numpy : 0.00027251243591308594 nb_pixel_total : 9736 time to create 1 rle with old method : 0.012548208236694336 length of segment : 170 time for calcul the mask position with numpy : 0.000492095947265625 nb_pixel_total : 11867 time to create 1 rle with old method : 0.014310359954833984 length of segment : 196 time for calcul the mask position with numpy : 0.004371166229248047 nb_pixel_total : 144502 time to create 1 rle with old method : 0.1691887378692627 length of segment : 630 time for calcul the mask position with numpy : 0.00026035308837890625 nb_pixel_total : 10356 time to create 1 rle with old method : 0.015073060989379883 length of segment : 93 time for calcul the mask position with numpy : 0.00025653839111328125 nb_pixel_total : 2542 time to create 1 rle with old method : 0.005454301834106445 length of segment : 64 time for calcul the mask position with numpy : 0.0002510547637939453 nb_pixel_total : 5112 time to create 1 rle with old method : 0.006394863128662109 length of segment : 93 time for calcul the mask position with numpy : 0.0003902912139892578 nb_pixel_total : 14246 time to create 1 rle with old method : 0.016823768615722656 length of segment : 117 time for calcul the mask position with numpy : 0.0029861927032470703 nb_pixel_total : 121907 time to create 1 rle with old method : 0.1621396541595459 length of segment : 387 time for calcul the mask position with numpy : 0.0005428791046142578 nb_pixel_total : 10451 time to create 1 rle with old method : 0.018270254135131836 length of segment : 115 time for calcul the mask position with numpy : 0.0003170967102050781 nb_pixel_total : 8574 time to create 1 rle with old method : 0.01769423484802246 length of segment : 154 time for calcul the mask position with numpy : 0.010735273361206055 nb_pixel_total : 378766 time to create 1 rle with new method : 0.018911361694335938 length of segment : 613 time for calcul the mask position with numpy : 0.0017817020416259766 nb_pixel_total : 59138 time to create 1 rle with old method : 0.07721590995788574 length of segment : 297 time for calcul the mask position with numpy : 0.0028312206268310547 nb_pixel_total : 122206 time to create 1 rle with old method : 0.14208436012268066 length of segment : 461 time for calcul the mask position with numpy : 0.001592397689819336 nb_pixel_total : 29523 time to create 1 rle with old method : 0.03637123107910156 length of segment : 322 time for calcul the mask position with numpy : 0.00023698806762695312 nb_pixel_total : 5940 time to create 1 rle with old method : 0.012037038803100586 length of segment : 85 time for calcul the mask position with numpy : 0.0001926422119140625 nb_pixel_total : 2001 time to create 1 rle with old method : 0.0042247772216796875 length of segment : 45 time for calcul the mask position with numpy : 0.0003147125244140625 nb_pixel_total : 5943 time to create 1 rle with old method : 0.012652158737182617 length of segment : 109 time for calcul the mask position with numpy : 0.00174713134765625 nb_pixel_total : 115303 time to create 1 rle with old method : 0.13681983947753906 length of segment : 369 time for calcul the mask position with numpy : 0.0014307498931884766 nb_pixel_total : 55310 time to create 1 rle with old method : 0.0657196044921875 length of segment : 256 time for calcul the mask position with numpy : 0.00024580955505371094 nb_pixel_total : 6521 time to create 1 rle with old method : 0.008387327194213867 length of segment : 87 time for calcul the mask position with numpy : 0.0017790794372558594 nb_pixel_total : 45777 time to create 1 rle with old method : 0.06160473823547363 length of segment : 377 time for calcul the mask position with numpy : 0.00029397010803222656 nb_pixel_total : 11331 time to create 1 rle with old method : 0.014519453048706055 length of segment : 130 time for calcul the mask position with numpy : 0.003927946090698242 nb_pixel_total : 168994 time to create 1 rle with new method : 0.009404659271240234 length of segment : 946 time for calcul the mask position with numpy : 0.0004696846008300781 nb_pixel_total : 11053 time to create 1 rle with old method : 0.0222928524017334 length of segment : 189 time for calcul the mask position with numpy : 0.0005216598510742188 nb_pixel_total : 11390 time to create 1 rle with old method : 0.013850688934326172 length of segment : 223 time for calcul the mask position with numpy : 0.00029969215393066406 nb_pixel_total : 7768 time to create 1 rle with old method : 0.010254859924316406 length of segment : 70 time for calcul the mask position with numpy : 0.0028684139251708984 nb_pixel_total : 71127 time to create 1 rle with old method : 0.08504557609558105 length of segment : 629 time for calcul the mask position with numpy : 0.0027549266815185547 nb_pixel_total : 107710 time to create 1 rle with old method : 0.13042855262756348 length of segment : 361 time for calcul the mask position with numpy : 0.0013506412506103516 nb_pixel_total : 45337 time to create 1 rle with old method : 0.05497407913208008 length of segment : 487 time for calcul the mask position with numpy : 0.002452373504638672 nb_pixel_total : 27002 time to create 1 rle with old method : 0.03236699104309082 length of segment : 302 time for calcul the mask position with numpy : 0.00013399124145507812 nb_pixel_total : 1934 time to create 1 rle with old method : 0.002393960952758789 length of segment : 56 time for calcul the mask position with numpy : 0.0006954669952392578 nb_pixel_total : 13585 time to create 1 rle with old method : 0.0162045955657959 length of segment : 229 time spent for convertir_results : 21.014989137649536 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 163 chid ids of type : 3663 Number RLEs to save : 54304 save missing photos in datou_result : time spend for datou_step_exec : 82.37444305419922 time spend to save output : 3.7345573902130127 total time spend for step 1 : 86.10900044441223 step2:crop_condition Tue Feb 11 21:17:58 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 163 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 15 About to insert : list_path_to_insert length 15 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 50 About to insert : list_path_to_insert length 50 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 12 About to insert : list_path_to_insert length 12 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1336797885, 1336797871, 1336797806, 1336797703, 1336796211, 1336796103, 1336794936, 1336794898, 1336794754, 1336794688] Looping around the photos to save general results len do output : 97 /-3670629389Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629384Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629391Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629409Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629421Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629431Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629443Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629483Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629478Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629488Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629381Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629368Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629398Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629402Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629411Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629426Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629458Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629451Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629452Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629456Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629470Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629464Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629475Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629495Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629506Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629400Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629417Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629434Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629472Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629486Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629492Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629485Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629366Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629371Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629375Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629363Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629380Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629378Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629376Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629385Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629404Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629403Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629394Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629392Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629406Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629414Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629415Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629412Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629405Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629408Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629416Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629423Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629419Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629424Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629433Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629430Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629435Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629444Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629445Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629447Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629455Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629474Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629461Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629473Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629468Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629471Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629484Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629489Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629494Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629479Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629481Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629480Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629496Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629515Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629511Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629510Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629505Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629504Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629513Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629520Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629522Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629519Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629370Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629372Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629390Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629393Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629396Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629453Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629463Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629490Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629500Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629502Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629507Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629517Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629437Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629442Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670629503Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797885', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797871', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797806', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797703', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796211', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796103', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794936', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794898', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794754', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794688', None, None, None, None, None, '2577749') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 301 time used for this insertion : 0.03412151336669922 save_final save missing photos in datou_result : time spend for datou_step_exec : 36.39388084411621 time spend to save output : 0.03694415092468262 total time spend for step 2 : 36.430824995040894 step3:thcl Tue Feb 11 21:18:35 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.018152475357055664 time to convert the images to numpy array : 0.10217833518981934 time to import caffe and check if the image exist : 0.00977778434753418 time to convert the images to numpy array : 0.13879966735839844 time to import caffe and check if the image exist : 0.01849985122680664 time to convert the images to numpy array : 0.16033291816711426 time to import caffe and check if the image exist : 0.014214277267456055 time to convert the images to numpy array : 0.16656970977783203 time to import caffe and check if the image exist : 0.02081918716430664 time to convert the images to numpy array : 0.18495798110961914 time to import caffe and check if the image exist : 0.01242208480834961 time to convert the images to numpy array : 0.20952916145324707 time to import caffe and check if the image exist : 0.017750978469848633 time to convert the images to numpy array : 0.20684146881103516 time to import caffe and check if the image exist : 0.013480901718139648 time to convert the images to numpy array : 0.21265220642089844 time to import caffe and check if the image exist : 0.016370534896850586 time to convert the images to numpy array : 0.21680474281311035 time to import caffe and check if the image exist : 0.020459651947021484 time to convert the images to numpy array : 0.21616816520690918 total time to convert the images to numpy array : 0.6908748149871826 list photo_ids error: [] list photo_ids correct : [-3670629522, -3670629519, -3670629370, -3670629372, -3670629390, -3670629393, -3670629396, -3670629453, -3670629463, -3670629490, -3670629389, -3670629384, -3670629391, -3670629409, -3670629421, -3670629431, -3670629443, -3670629483, -3670629478, -3670629488, -3670629492, -3670629485, -3670629366, -3670629371, -3670629375, -3670629363, -3670629380, -3670629378, -3670629376, -3670629385, -3670629381, -3670629368, -3670629398, -3670629402, -3670629411, -3670629426, -3670629458, -3670629451, -3670629452, -3670629456, -3670629470, -3670629464, -3670629475, -3670629495, -3670629506, -3670629400, -3670629417, -3670629434, -3670629472, -3670629486, -3670629481, -3670629480, -3670629496, -3670629515, -3670629511, -3670629510, -3670629505, -3670629504, -3670629513, -3670629520, -3670629500, -3670629502, -3670629507, -3670629517, -3670629437, -3670629442, -3670629503, -3670629455, -3670629474, -3670629461, -3670629473, -3670629468, -3670629471, -3670629484, -3670629489, -3670629494, -3670629479, -3670629404, -3670629403, -3670629394, -3670629392, -3670629406, -3670629414, -3670629415, -3670629412, -3670629405, -3670629408, -3670629416, -3670629423, -3670629419, -3670629424, -3670629433, -3670629430, -3670629435, -3670629444, -3670629445, -3670629447] number of photos to traite : 97 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10332 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10113 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.0101349353790283 time used to do the prediction : 0.2568836212158203 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10111 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10111 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.0785977840423584 time used to do the prediction : 0.17934560775756836 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.8371810913085938e-05 save missing photos in datou_result : time spend for datou_step_exec : 10.095197439193726 time spend to save output : 0.0006799697875976562 total time spend for step 3 : 10.095877408981323 step4:merge_mask_thcl_custom Tue Feb 11 21:18:45 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 163 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1336797885, 1336797871, 1336797806, 1336797703, 1336796211, 1336796103, 1336794936, 1336794898, 1336794754, 1336794688] Looping around the photos to save general results len do output : 10 /1336797885Didn't retrieve data .Didn't retrieve data . /1336797871Didn't retrieve data .Didn't retrieve data . /1336797806Didn't retrieve data .Didn't retrieve data . /1336797703Didn't retrieve data .Didn't retrieve data . /1336796211Didn't retrieve data .Didn't retrieve data . /1336796103Didn't retrieve data .Didn't retrieve data . /1336794936Didn't retrieve data .Didn't retrieve data . /1336794898Didn't retrieve data .Didn't retrieve data . /1336794754Didn't retrieve data .Didn't retrieve data . /1336794688Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797885', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797871', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797806', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797703', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796211', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796103', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794936', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794898', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794754', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794688', None, None, None, None, None, '2577749') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014469623565673828 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.061109066009521484 time spend to save output : 0.015242576599121094 total time spend for step 4 : 0.07635164260864258 step5:rle_unique_nms_with_priority Tue Feb 11 21:18:45 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 21 nb_hashtags : 6 time to prepare the origin masks : 9.645680665969849 time for calcul the mask position with numpy : 0.20115375518798828 nb_pixel_total : 5769261 time to create 1 rle with new method : 0.3432443141937256 time for calcul the mask position with numpy : 0.02505350112915039 nb_pixel_total : 9970 time to create 1 rle with old method : 0.011058330535888672 time for calcul the mask position with numpy : 0.02343297004699707 nb_pixel_total : 39186 time to create 1 rle with old method : 0.044222354888916016 time for calcul the mask position with numpy : 0.025433778762817383 nb_pixel_total : 1385 time to create 1 rle with old method : 0.0018627643585205078 time for calcul the mask position with numpy : 0.026866674423217773 nb_pixel_total : 109073 time to create 1 rle with old method : 0.1278228759765625 time for calcul the mask position with numpy : 0.023589611053466797 nb_pixel_total : 58081 time to create 1 rle with old method : 0.06235051155090332 time for calcul the mask position with numpy : 0.02471017837524414 nb_pixel_total : 4256 time to create 1 rle with old method : 0.007174491882324219 time for calcul the mask position with numpy : 0.02548837661743164 nb_pixel_total : 16306 time to create 1 rle with old method : 0.026967287063598633 time for calcul the mask position with numpy : 0.02474188804626465 nb_pixel_total : 7173 time to create 1 rle with old method : 0.008206844329833984 time for calcul the mask position with numpy : 0.02398991584777832 nb_pixel_total : 16829 time to create 1 rle with old method : 0.01920318603515625 time for calcul the mask position with numpy : 0.023916006088256836 nb_pixel_total : 2734 time to create 1 rle with old method : 0.0033385753631591797 time for calcul the mask position with numpy : 0.02509474754333496 nb_pixel_total : 5355 time to create 1 rle with old method : 0.006377458572387695 time for calcul the mask position with numpy : 0.02703118324279785 nb_pixel_total : 151040 time to create 1 rle with new method : 0.3164680004119873 time for calcul the mask position with numpy : 0.03310227394104004 nb_pixel_total : 36481 time to create 1 rle with old method : 0.03892803192138672 time for calcul the mask position with numpy : 0.03706979751586914 nb_pixel_total : 11626 time to create 1 rle with old method : 0.012833118438720703 time for calcul the mask position with numpy : 0.04133319854736328 nb_pixel_total : 315994 time to create 1 rle with new method : 0.34934306144714355 time for calcul the mask position with numpy : 0.03945207595825195 nb_pixel_total : 68406 time to create 1 rle with old method : 0.07702851295471191 time for calcul the mask position with numpy : 0.04062199592590332 nb_pixel_total : 21719 time to create 1 rle with old method : 0.024440526962280273 time for calcul the mask position with numpy : 0.039345502853393555 nb_pixel_total : 6368 time to create 1 rle with old method : 0.007451534271240234 time for calcul the mask position with numpy : 0.03949594497680664 nb_pixel_total : 298176 time to create 1 rle with new method : 0.32804250717163086 time for calcul the mask position with numpy : 0.03754901885986328 nb_pixel_total : 6439 time to create 1 rle with old method : 0.006988048553466797 time for calcul the mask position with numpy : 0.03818535804748535 nb_pixel_total : 94382 time to create 1 rle with old method : 0.10434103012084961 create new chi : 2.871978521347046 time to delete rle : 0.017058849334716797 batch 1 Loaded 22 chid ids of type : 3726 Number RLEs to save : 13458 TO DO : save crop sub photo not yet done ! save time : 0.7403407096862793 nb_obj : 22 nb_hashtags : 6 time to prepare the origin masks : 10.517466068267822 time for calcul the mask position with numpy : 0.30438995361328125 nb_pixel_total : 5999728 time to create 1 rle with new method : 0.3154294490814209 time for calcul the mask position with numpy : 0.023894786834716797 nb_pixel_total : 17502 time to create 1 rle with old method : 0.020373106002807617 time for calcul the mask position with numpy : 0.035869598388671875 nb_pixel_total : 10540 time to create 1 rle with old method : 0.012463092803955078 time for calcul the mask position with numpy : 0.04434847831726074 nb_pixel_total : 25149 time to create 1 rle with old method : 0.03389406204223633 time for calcul the mask position with numpy : 0.03747844696044922 nb_pixel_total : 52776 time to create 1 rle with old method : 0.06401610374450684 time for calcul the mask position with numpy : 0.037217140197753906 nb_pixel_total : 55930 time to create 1 rle with old method : 0.0659797191619873 time for calcul the mask position with numpy : 0.035727739334106445 nb_pixel_total : 34462 time to create 1 rle with old method : 0.0394749641418457 time for calcul the mask position with numpy : 0.03572416305541992 nb_pixel_total : 6120 time to create 1 rle with old method : 0.006989240646362305 time for calcul the mask position with numpy : 0.039017438888549805 nb_pixel_total : 189940 time to create 1 rle with new method : 0.31763553619384766 time for calcul the mask position with numpy : 0.04058194160461426 nb_pixel_total : 62999 time to create 1 rle with old method : 0.07492589950561523 time for calcul the mask position with numpy : 0.02527928352355957 nb_pixel_total : 2515 time to create 1 rle with old method : 0.003184080123901367 time for calcul the mask position with numpy : 0.025151729583740234 nb_pixel_total : 352232 time to create 1 rle with new method : 0.31307315826416016 time for calcul the mask position with numpy : 0.022841215133666992 nb_pixel_total : 10839 time to create 1 rle with old method : 0.012315511703491211 time for calcul the mask position with numpy : 0.02165961265563965 nb_pixel_total : 65424 time to create 1 rle with old method : 0.0777435302734375 time for calcul the mask position with numpy : 0.0222623348236084 nb_pixel_total : 3733 time to create 1 rle with old method : 0.0044727325439453125 time for calcul the mask position with numpy : 0.021246910095214844 nb_pixel_total : 9592 time to create 1 rle with old method : 0.010994195938110352 time for calcul the mask position with numpy : 0.02080082893371582 nb_pixel_total : 2873 time to create 1 rle with old method : 0.003592967987060547 time for calcul the mask position with numpy : 0.0210874080657959 nb_pixel_total : 2788 time to create 1 rle with old method : 0.0032012462615966797 time for calcul the mask position with numpy : 0.021386384963989258 nb_pixel_total : 4932 time to create 1 rle with old method : 0.005637645721435547 time for calcul the mask position with numpy : 0.020773649215698242 nb_pixel_total : 9698 time to create 1 rle with old method : 0.011064291000366211 time for calcul the mask position with numpy : 0.02157115936279297 nb_pixel_total : 43158 time to create 1 rle with old method : 0.048647165298461914 time for calcul the mask position with numpy : 0.02410578727722168 nb_pixel_total : 10637 time to create 1 rle with old method : 0.012045621871948242 time for calcul the mask position with numpy : 0.03510284423828125 nb_pixel_total : 76673 time to create 1 rle with old method : 0.09590697288513184 create new chi : 2.5760130882263184 time to delete rle : 0.0012454986572265625 batch 1 Loaded 23 chid ids of type : 3726 Number RLEs to save : 12834 TO DO : save crop sub photo not yet done ! save time : 0.7187111377716064 nb_obj : 13 nb_hashtags : 4 time to prepare the origin masks : 5.863414525985718 time for calcul the mask position with numpy : 0.4583287239074707 nb_pixel_total : 4410393 time to create 1 rle with new method : 0.3836510181427002 time for calcul the mask position with numpy : 0.022195100784301758 nb_pixel_total : 29733 time to create 1 rle with old method : 0.031077146530151367 time for calcul the mask position with numpy : 0.02196025848388672 nb_pixel_total : 27122 time to create 1 rle with old method : 0.03496551513671875 time for calcul the mask position with numpy : 0.02507805824279785 nb_pixel_total : 370176 time to create 1 rle with new method : 0.3846559524536133 time for calcul the mask position with numpy : 0.03739452362060547 nb_pixel_total : 1838 time to create 1 rle with old method : 0.0021047592163085938 time for calcul the mask position with numpy : 0.036942243576049805 nb_pixel_total : 13 time to create 1 rle with old method : 0.00012159347534179688 time for calcul the mask position with numpy : 0.03810882568359375 nb_pixel_total : 5142 time to create 1 rle with old method : 0.005769252777099609 time for calcul the mask position with numpy : 0.03677248954772949 nb_pixel_total : 21174 time to create 1 rle with old method : 0.022098064422607422 time for calcul the mask position with numpy : 0.03686380386352539 nb_pixel_total : 70375 time to create 1 rle with old method : 0.07314085960388184 time for calcul the mask position with numpy : 0.053855180740356445 nb_pixel_total : 1892334 time to create 1 rle with new method : 0.3730630874633789 time for calcul the mask position with numpy : 0.037389516830444336 nb_pixel_total : 19813 time to create 1 rle with old method : 0.021647930145263672 time for calcul the mask position with numpy : 0.037732839584350586 nb_pixel_total : 1541 time to create 1 rle with old method : 0.002051115036010742 time for calcul the mask position with numpy : 0.037694454193115234 nb_pixel_total : 46690 time to create 1 rle with old method : 0.05247306823730469 time for calcul the mask position with numpy : 0.03969120979309082 nb_pixel_total : 153896 time to create 1 rle with new method : 0.31607866287231445 create new chi : 2.7387712001800537 time to delete rle : 0.001809835433959961 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 13522 TO DO : save crop sub photo not yet done ! save time : 1.1080758571624756 nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 3.8493103981018066 time for calcul the mask position with numpy : 0.12979984283447266 nb_pixel_total : 2954748 time to create 1 rle with new method : 0.43480563163757324 time for calcul the mask position with numpy : 0.02382349967956543 nb_pixel_total : 15345 time to create 1 rle with old method : 0.017798185348510742 time for calcul the mask position with numpy : 0.023195981979370117 nb_pixel_total : 2571 time to create 1 rle with old method : 0.003014802932739258 time for calcul the mask position with numpy : 0.023792505264282227 nb_pixel_total : 93794 time to create 1 rle with old method : 0.10242533683776855 time for calcul the mask position with numpy : 0.02467942237854004 nb_pixel_total : 75650 time to create 1 rle with old method : 0.0844419002532959 time for calcul the mask position with numpy : 0.02360987663269043 nb_pixel_total : 48420 time to create 1 rle with old method : 0.05443263053894043 time for calcul the mask position with numpy : 0.4003164768218994 nb_pixel_total : 3827520 time to create 1 rle with new method : 0.6839172840118408 time for calcul the mask position with numpy : 0.026382923126220703 nb_pixel_total : 32192 time to create 1 rle with old method : 0.03476214408874512 create new chi : 2.1610777378082275 time to delete rle : 0.00110626220703125 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 12205 TO DO : save crop sub photo not yet done ! save time : 0.6915748119354248 nb_obj : 14 nb_hashtags : 7 time to prepare the origin masks : 6.209879159927368 time for calcul the mask position with numpy : 0.31847548484802246 nb_pixel_total : 6429733 time to create 1 rle with new method : 0.323702335357666 time for calcul the mask position with numpy : 0.02328777313232422 nb_pixel_total : 111753 time to create 1 rle with old method : 0.11684727668762207 time for calcul the mask position with numpy : 0.021953105926513672 nb_pixel_total : 12644 time to create 1 rle with old method : 0.013681173324584961 time for calcul the mask position with numpy : 0.023651599884033203 nb_pixel_total : 8171 time to create 1 rle with old method : 0.00894474983215332 time for calcul the mask position with numpy : 0.023590803146362305 nb_pixel_total : 9031 time to create 1 rle with old method : 0.010081768035888672 time for calcul the mask position with numpy : 0.02571415901184082 nb_pixel_total : 69506 time to create 1 rle with old method : 0.07176566123962402 time for calcul the mask position with numpy : 0.02268671989440918 nb_pixel_total : 18070 time to create 1 rle with old method : 0.019228696823120117 time for calcul the mask position with numpy : 0.0221707820892334 nb_pixel_total : 795 time to create 1 rle with old method : 0.0011048316955566406 time for calcul the mask position with numpy : 0.0230712890625 nb_pixel_total : 38406 time to create 1 rle with old method : 0.03994107246398926 time for calcul the mask position with numpy : 0.022703170776367188 nb_pixel_total : 87425 time to create 1 rle with old method : 0.09026741981506348 time for calcul the mask position with numpy : 0.021767139434814453 nb_pixel_total : 2263 time to create 1 rle with old method : 0.0025446414947509766 time for calcul the mask position with numpy : 0.021662235260009766 nb_pixel_total : 6259 time to create 1 rle with old method : 0.006615161895751953 time for calcul the mask position with numpy : 0.023319244384765625 nb_pixel_total : 245398 time to create 1 rle with new method : 0.3120443820953369 time for calcul the mask position with numpy : 0.025635957717895508 nb_pixel_total : 2813 time to create 1 rle with old method : 0.0033333301544189453 time for calcul the mask position with numpy : 0.022884130477905273 nb_pixel_total : 7973 time to create 1 rle with old method : 0.008840799331665039 create new chi : 1.7330777645111084 time to delete rle : 0.0009083747863769531 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 8690 TO DO : save crop sub photo not yet done ! save time : 0.6089558601379395 nb_obj : 17 nb_hashtags : 6 time to prepare the origin masks : 7.146541357040405 time for calcul the mask position with numpy : 0.2461695671081543 nb_pixel_total : 6418743 time to create 1 rle with new method : 0.32613277435302734 time for calcul the mask position with numpy : 0.02432084083557129 nb_pixel_total : 64029 time to create 1 rle with old method : 0.07197880744934082 time for calcul the mask position with numpy : 0.03600668907165527 nb_pixel_total : 101921 time to create 1 rle with old method : 0.1148681640625 time for calcul the mask position with numpy : 0.035131216049194336 nb_pixel_total : 45695 time to create 1 rle with old method : 0.051389455795288086 time for calcul the mask position with numpy : 0.03476285934448242 nb_pixel_total : 9250 time to create 1 rle with old method : 0.010829448699951172 time for calcul the mask position with numpy : 0.035120487213134766 nb_pixel_total : 45514 time to create 1 rle with old method : 0.0547940731048584 time for calcul the mask position with numpy : 0.03585004806518555 nb_pixel_total : 3383 time to create 1 rle with old method : 0.004110097885131836 time for calcul the mask position with numpy : 0.033731937408447266 nb_pixel_total : 7251 time to create 1 rle with old method : 0.008243083953857422 time for calcul the mask position with numpy : 0.03483939170837402 nb_pixel_total : 1596 time to create 1 rle with old method : 0.002012014389038086 time for calcul the mask position with numpy : 0.03372550010681152 nb_pixel_total : 3221 time to create 1 rle with old method : 0.003811359405517578 time for calcul the mask position with numpy : 0.033951520919799805 nb_pixel_total : 53165 time to create 1 rle with old method : 0.05997014045715332 time for calcul the mask position with numpy : 0.03360247611999512 nb_pixel_total : 15347 time to create 1 rle with old method : 0.017269134521484375 time for calcul the mask position with numpy : 0.02623295783996582 nb_pixel_total : 1249 time to create 1 rle with old method : 0.0015590190887451172 time for calcul the mask position with numpy : 0.02068185806274414 nb_pixel_total : 34855 time to create 1 rle with old method : 0.038913726806640625 time for calcul the mask position with numpy : 0.022653579711914062 nb_pixel_total : 189042 time to create 1 rle with new method : 0.32993078231811523 time for calcul the mask position with numpy : 0.020859479904174805 nb_pixel_total : 12272 time to create 1 rle with old method : 0.014106988906860352 time for calcul the mask position with numpy : 0.020627975463867188 nb_pixel_total : 31148 time to create 1 rle with old method : 0.03481101989746094 time for calcul the mask position with numpy : 0.0206756591796875 nb_pixel_total : 12559 time to create 1 rle with old method : 0.014566421508789062 create new chi : 1.9720232486724854 time to delete rle : 0.0013234615325927734 batch 1 Loaded 18 chid ids of type : 3726 Number RLEs to save : 9965 TO DO : save crop sub photo not yet done ! save time : 0.5934112071990967 nb_obj : 19 nb_hashtags : 6 time to prepare the origin masks : 8.68217396736145 time for calcul the mask position with numpy : 0.2179872989654541 nb_pixel_total : 4209812 time to create 1 rle with new method : 0.3798251152038574 time for calcul the mask position with numpy : 0.022634506225585938 nb_pixel_total : 1614 time to create 1 rle with old method : 0.0020437240600585938 time for calcul the mask position with numpy : 0.024670124053955078 nb_pixel_total : 135657 time to create 1 rle with old method : 0.14601707458496094 time for calcul the mask position with numpy : 0.029941082000732422 nb_pixel_total : 7230 time to create 1 rle with old method : 0.007735013961791992 time for calcul the mask position with numpy : 0.024483919143676758 nb_pixel_total : 51222 time to create 1 rle with old method : 0.05788707733154297 time for calcul the mask position with numpy : 0.02543473243713379 nb_pixel_total : 1005 time to create 1 rle with old method : 0.0013370513916015625 time for calcul the mask position with numpy : 0.042204856872558594 nb_pixel_total : 82767 time to create 1 rle with old method : 0.09907031059265137 time for calcul the mask position with numpy : 0.05674266815185547 nb_pixel_total : 511995 time to create 1 rle with new method : 0.34488964080810547 time for calcul the mask position with numpy : 0.03539705276489258 nb_pixel_total : 2998 time to create 1 rle with old method : 0.0036644935607910156 time for calcul the mask position with numpy : 0.03713059425354004 nb_pixel_total : 3835 time to create 1 rle with old method : 0.004677295684814453 time for calcul the mask position with numpy : 0.03826498985290527 nb_pixel_total : 53565 time to create 1 rle with old method : 0.06009721755981445 time for calcul the mask position with numpy : 0.03672170639038086 nb_pixel_total : 4637 time to create 1 rle with old method : 0.005556821823120117 time for calcul the mask position with numpy : 0.03783822059631348 nb_pixel_total : 117692 time to create 1 rle with old method : 0.13822555541992188 time for calcul the mask position with numpy : 0.03734230995178223 nb_pixel_total : 90426 time to create 1 rle with old method : 0.10108137130737305 time for calcul the mask position with numpy : 0.0382838249206543 nb_pixel_total : 152605 time to create 1 rle with new method : 0.3157081604003906 time for calcul the mask position with numpy : 0.036951541900634766 nb_pixel_total : 2220 time to create 1 rle with old method : 0.0026695728302001953 time for calcul the mask position with numpy : 0.03883767127990723 nb_pixel_total : 6155 time to create 1 rle with old method : 0.007422447204589844 time for calcul the mask position with numpy : 0.057341575622558594 nb_pixel_total : 1498473 time to create 1 rle with new method : 0.3309156894683838 time for calcul the mask position with numpy : 0.0400543212890625 nb_pixel_total : 1656 time to create 1 rle with old method : 0.002798795700073242 time for calcul the mask position with numpy : 0.04295969009399414 nb_pixel_total : 114676 time to create 1 rle with old method : 0.12752223014831543 create new chi : 3.176398754119873 time to delete rle : 0.0023818016052246094 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 17588 TO DO : save crop sub photo not yet done ! save time : 0.9749267101287842 nb_obj : 18 nb_hashtags : 6 time to prepare the origin masks : 8.183002948760986 time for calcul the mask position with numpy : 0.3561861515045166 nb_pixel_total : 6396319 time to create 1 rle with new method : 0.7058148384094238 time for calcul the mask position with numpy : 0.024135351181030273 nb_pixel_total : 10241 time to create 1 rle with old method : 0.012011528015136719 time for calcul the mask position with numpy : 0.03967785835266113 nb_pixel_total : 46363 time to create 1 rle with old method : 0.051917076110839844 time for calcul the mask position with numpy : 0.03631114959716797 nb_pixel_total : 11332 time to create 1 rle with old method : 0.01370096206665039 time for calcul the mask position with numpy : 0.035948991775512695 nb_pixel_total : 71490 time to create 1 rle with old method : 0.08350586891174316 time for calcul the mask position with numpy : 0.035817623138427734 nb_pixel_total : 9976 time to create 1 rle with old method : 0.012928009033203125 time for calcul the mask position with numpy : 0.0540773868560791 nb_pixel_total : 137242 time to create 1 rle with old method : 0.17980170249938965 time for calcul the mask position with numpy : 0.041162967681884766 nb_pixel_total : 11861 time to create 1 rle with old method : 0.013489007949829102 time for calcul the mask position with numpy : 0.02630305290222168 nb_pixel_total : 8410 time to create 1 rle with old method : 0.009943962097167969 time for calcul the mask position with numpy : 0.03361630439758301 nb_pixel_total : 22178 time to create 1 rle with old method : 0.025005578994750977 time for calcul the mask position with numpy : 0.03379988670349121 nb_pixel_total : 17031 time to create 1 rle with old method : 0.01927042007446289 time for calcul the mask position with numpy : 0.033104896545410156 nb_pixel_total : 9526 time to create 1 rle with old method : 0.012775897979736328 time for calcul the mask position with numpy : 0.021021127700805664 nb_pixel_total : 21004 time to create 1 rle with old method : 0.023717641830444336 time for calcul the mask position with numpy : 0.021307706832885742 nb_pixel_total : 53 time to create 1 rle with old method : 0.0002243518829345703 time for calcul the mask position with numpy : 0.02162957191467285 nb_pixel_total : 143640 time to create 1 rle with old method : 0.1580812931060791 time for calcul the mask position with numpy : 0.02076435089111328 nb_pixel_total : 56264 time to create 1 rle with old method : 0.062096595764160156 time for calcul the mask position with numpy : 0.02689385414123535 nb_pixel_total : 2868 time to create 1 rle with old method : 0.0039730072021484375 time for calcul the mask position with numpy : 0.02154231071472168 nb_pixel_total : 54171 time to create 1 rle with old method : 0.062341928482055664 time for calcul the mask position with numpy : 0.021075010299682617 nb_pixel_total : 20271 time to create 1 rle with old method : 0.03014087677001953 create new chi : 2.4396650791168213 time to delete rle : 0.0011456012725830078 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 11338 TO DO : save crop sub photo not yet done ! save time : 0.6418559551239014 nb_obj : 18 nb_hashtags : 6 time to prepare the origin masks : 8.229156255722046 time for calcul the mask position with numpy : 0.31119799613952637 nb_pixel_total : 6017449 time to create 1 rle with new method : 0.6228997707366943 time for calcul the mask position with numpy : 0.021633386611938477 nb_pixel_total : 28532 time to create 1 rle with old method : 0.03203248977661133 time for calcul the mask position with numpy : 0.02258014678955078 nb_pixel_total : 122118 time to create 1 rle with old method : 0.14435076713562012 time for calcul the mask position with numpy : 0.022619962692260742 nb_pixel_total : 59002 time to create 1 rle with old method : 0.0771639347076416 time for calcul the mask position with numpy : 0.028337717056274414 nb_pixel_total : 378187 time to create 1 rle with new method : 0.36414480209350586 time for calcul the mask position with numpy : 0.021386384963989258 nb_pixel_total : 5885 time to create 1 rle with old method : 0.007049083709716797 time for calcul the mask position with numpy : 0.021529674530029297 nb_pixel_total : 2678 time to create 1 rle with old method : 0.003799915313720703 time for calcul the mask position with numpy : 0.02109217643737793 nb_pixel_total : 7862 time to create 1 rle with old method : 0.009432077407836914 time for calcul the mask position with numpy : 0.021494626998901367 nb_pixel_total : 3271 time to create 1 rle with old method : 0.0037572383880615234 time for calcul the mask position with numpy : 0.021089553833007812 nb_pixel_total : 5929 time to create 1 rle with old method : 0.006853342056274414 time for calcul the mask position with numpy : 0.021428823471069336 nb_pixel_total : 115002 time to create 1 rle with old method : 0.1280076503753662 time for calcul the mask position with numpy : 0.020572900772094727 nb_pixel_total : 1990 time to create 1 rle with old method : 0.002292633056640625 time for calcul the mask position with numpy : 0.020888090133666992 nb_pixel_total : 45640 time to create 1 rle with old method : 0.0518648624420166 time for calcul the mask position with numpy : 0.02113032341003418 nb_pixel_total : 10406 time to create 1 rle with old method : 0.011838197708129883 time for calcul the mask position with numpy : 0.02163839340209961 nb_pixel_total : 14225 time to create 1 rle with old method : 0.017289161682128906 time for calcul the mask position with numpy : 0.023107051849365234 nb_pixel_total : 169157 time to create 1 rle with new method : 0.37215495109558105 time for calcul the mask position with numpy : 0.021158456802368164 nb_pixel_total : 5094 time to create 1 rle with old method : 0.006060600280761719 time for calcul the mask position with numpy : 0.02179884910583496 nb_pixel_total : 2517 time to create 1 rle with old method : 0.003818988800048828 time for calcul the mask position with numpy : 0.023667335510253906 nb_pixel_total : 55296 time to create 1 rle with old method : 0.06297826766967773 create new chi : 2.7282910346984863 time to delete rle : 0.0015511512756347656 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 11322 TO DO : save crop sub photo not yet done ! save time : 0.6201217174530029 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 6.977075099945068 time for calcul the mask position with numpy : 0.24673962593078613 nb_pixel_total : 6752049 time to create 1 rle with new method : 0.32152295112609863 time for calcul the mask position with numpy : 0.02076268196105957 nb_pixel_total : 13558 time to create 1 rle with old method : 0.015455961227416992 time for calcul the mask position with numpy : 0.02153468132019043 nb_pixel_total : 1925 time to create 1 rle with old method : 0.0022873878479003906 time for calcul the mask position with numpy : 0.021162033081054688 nb_pixel_total : 47700 time to create 1 rle with old method : 0.05473661422729492 time for calcul the mask position with numpy : 0.021006107330322266 nb_pixel_total : 11355 time to create 1 rle with old method : 0.013497114181518555 time for calcul the mask position with numpy : 0.021944761276245117 nb_pixel_total : 70998 time to create 1 rle with old method : 0.07996129989624023 time for calcul the mask position with numpy : 0.02208232879638672 nb_pixel_total : 7615 time to create 1 rle with old method : 0.008896589279174805 time for calcul the mask position with numpy : 0.021610021591186523 nb_pixel_total : 11019 time to create 1 rle with old method : 0.012540578842163086 time for calcul the mask position with numpy : 0.020566940307617188 nb_pixel_total : 26786 time to create 1 rle with old method : 0.030178070068359375 time for calcul the mask position with numpy : 0.021094799041748047 nb_pixel_total : 107235 time to create 1 rle with old method : 0.11925816535949707 create new chi : 1.1373653411865234 time to delete rle : 0.0007882118225097656 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 7114 TO DO : save crop sub photo not yet done ! save time : 0.4603385925292969 map_output_result : {1336797885: (0.0, 'Should be the crop_list due to order', 0.0), 1336797871: (0.0, 'Should be the crop_list due to order', 0.0), 1336797806: (0.0, 'Should be the crop_list due to order', 0.0), 1336797703: (0.0, 'Should be the crop_list due to order', 0.0), 1336796211: (0.0, 'Should be the crop_list due to order', 0.0), 1336796103: (0.0, 'Should be the crop_list due to order', 0.0), 1336794936: (0.0, 'Should be the crop_list due to order', 0.0), 1336794898: (0.0, 'Should be the crop_list due to order', 0.0), 1336794754: (0.0, 'Should be the crop_list due to order', 0.0), 1336794688: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1336797885, 1336797871, 1336797806, 1336797703, 1336796211, 1336796103, 1336794936, 1336794898, 1336794754, 1336794688] Looping around the photos to save general results len do output : 10 /1336797885.Didn't retrieve data . /1336797871.Didn't retrieve data . /1336797806.Didn't retrieve data . /1336797703.Didn't retrieve data . /1336796211.Didn't retrieve data . /1336796103.Didn't retrieve data . /1336794936.Didn't retrieve data . /1336794898.Didn't retrieve data . /1336794754.Didn't retrieve data . /1336794688.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797885', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797871', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797806', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797703', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796211', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796103', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794936', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794898', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794754', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794688', None, None, None, None, None, '2577749') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013209104537963867 save_final save missing photos in datou_result : time spend for datou_step_exec : 107.11454343795776 time spend to save output : 0.013682126998901367 total time spend for step 5 : 107.12822556495667 step6:crop_condition Tue Feb 11 21:20:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 168 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305235_32503 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.5021471977233887 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305241_32503 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.5844745635986328 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305244_32503 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9547874927520752 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305246_32503 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5893785953521729 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! About to upload 10 photos upload in portfolio : 4869462 init cache_photo without model_param we have 10 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305249_32503 we have uploaded 10 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.380190372467041 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305254_32503 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.823021411895752 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305256_32503 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8625648021697998 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1336797885, 1336797871, 1336797806, 1336797703, 1336796211, 1336796103, 1336794936, 1336794898, 1336794754, 1336794688] Looping around the photos to save general results len do output : 28 /1336909847Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909848Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909849Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909850Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909851Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909888Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909892Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909896Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909900Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909904Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909906Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909909Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909910Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909911Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909915Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909916Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909917Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909918Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909919Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909920Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909921Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909922Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909923Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909924Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909926Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909927Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909930Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909931Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797885', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797871', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797806', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797703', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796211', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796103', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794936', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794898', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794754', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794688', None, None, None, None, None, '2577749') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 94 time used for this insertion : 0.030482053756713867 save_final save missing photos in datou_result : time spend for datou_step_exec : 25.763400316238403 time spend to save output : 0.03137660026550293 total time spend for step 6 : 25.794776916503906 step7:ventilate_hashtags_in_portfolio Tue Feb 11 21:20:58 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20461697 get user id for portfolio 20461697 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461697 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_gris','flou','Teint_Dans_La_Masse','kraft','Carton_brun','cartonnette','autre_refus','environnement','papier','mal_croppe','metal','plastique')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461697 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_gris','flou','Teint_Dans_La_Masse','kraft','Carton_brun','cartonnette','autre_refus','environnement','papier','mal_croppe','metal','plastique')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461697 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_gris','flou','Teint_Dans_La_Masse','kraft','Carton_brun','cartonnette','autre_refus','environnement','papier','mal_croppe','metal','plastique')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20461978,20461979,20461980,20461981,20461982,20461983,20461984,20461985,20461986,20461987,20461988,20461989?tags=Carton_gris,flou,Teint_Dans_La_Masse,kraft,Carton_brun,cartonnette,autre_refus,environnement,papier,mal_croppe,metal,plastique Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1336797885, 1336797871, 1336797806, 1336797703, 1336796211, 1336796103, 1336794936, 1336794898, 1336794754, 1336794688] Looping around the photos to save general results len do output : 1 /20461697. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797885', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797871', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797806', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797703', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796211', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796103', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794936', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794898', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794754', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794688', None, None, None, None, None, '2577749') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.015374422073364258 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.6879546642303467 time spend to save output : 0.01566767692565918 total time spend for step 7 : 1.7036223411560059 step8:final Tue Feb 11 21:20:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1336797885: ('0.08017768812741913',), 1336797871: ('0.08017768812741913',), 1336797806: ('0.08017768812741913',), 1336797703: ('0.08017768812741913',), 1336796211: ('0.08017768812741913',), 1336796103: ('0.08017768812741913',), 1336794936: ('0.08017768812741913',), 1336794898: ('0.08017768812741913',), 1336794754: ('0.08017768812741913',), 1336794688: ('0.08017768812741913',)} new output for save of step final : {1336797885: ('0.08017768812741913',), 1336797871: ('0.08017768812741913',), 1336797806: ('0.08017768812741913',), 1336797703: ('0.08017768812741913',), 1336796211: ('0.08017768812741913',), 1336796103: ('0.08017768812741913',), 1336794936: ('0.08017768812741913',), 1336794898: ('0.08017768812741913',), 1336794754: ('0.08017768812741913',), 1336794688: ('0.08017768812741913',)} [1336797885, 1336797871, 1336797806, 1336797703, 1336796211, 1336796103, 1336794936, 1336794898, 1336794754, 1336794688] Looping around the photos to save general results len do output : 10 /1336797885.Didn't retrieve data . /1336797871.Didn't retrieve data . /1336797806.Didn't retrieve data . /1336797703.Didn't retrieve data . /1336796211.Didn't retrieve data . /1336796103.Didn't retrieve data . /1336794936.Didn't retrieve data . /1336794898.Didn't retrieve data . /1336794754.Didn't retrieve data . /1336794688.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797885', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797871', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797806', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797703', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796211', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796103', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794936', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794898', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794754', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794688', None, None, None, None, None, '2577749') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013594865798950195 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.09690213203430176 time spend to save output : 0.014336585998535156 total time spend for step 8 : 0.11123871803283691 step9:velours_tree Tue Feb 11 21:21:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.11679792404174805 time spend to save output : 3.838539123535156e-05 total time spend for step 9 : 0.1168363094329834 step10:send_mail_cod Tue Feb 11 21:21:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20461697_11-02-2025_21_21_00.pdf 20461979 imagette204619791739305260 20461981 change filename to text .change filename to text .imagette204619811739305260 20461983 change filename to text .imagette204619831739305260 20461984 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204619841739305260 20461987 imagette204619871739305260 20461988 imagette204619881739305260 20461989 change filename to text .change filename to text .imagette204619891739305260 20461978 change filename to text .change filename to text .imagette204619781739305260 20461980 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204619801739305261 20461982 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204619821739305261 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20461697 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20461978,20461979,20461980,20461981,20461982,20461983,20461984,20461985,20461986,20461987,20461988,20461989?tags=Carton_gris,flou,Teint_Dans_La_Masse,kraft,Carton_brun,cartonnette,autre_refus,environnement,papier,mal_croppe,metal,plastique your option no_mail is active, we will not send the real mail to your client args[1336797885] : ('0.08017768812741913',) no score found for photo 1336797885 apple ('0.08017768812741913',) We are sending mail with results at report@fotonower.com args[1336797871] : ('0.08017768812741913',) no score found for photo 1336797871 apple ('0.08017768812741913',) We are sending mail with results at report@fotonower.com args[1336797806] : ('0.08017768812741913',) no score found for photo 1336797806 apple ('0.08017768812741913',) We are sending mail with results at report@fotonower.com args[1336797703] : ('0.08017768812741913',) no score found for photo 1336797703 apple ('0.08017768812741913',) We are sending mail with results at report@fotonower.com args[1336796211] : ('0.08017768812741913',) no score found for photo 1336796211 apple ('0.08017768812741913',) We are sending mail with results at report@fotonower.com args[1336796103] : ('0.08017768812741913',) no score found for photo 1336796103 apple ('0.08017768812741913',) We are sending mail with results at report@fotonower.com args[1336794936] : ('0.08017768812741913',) no score found for photo 1336794936 apple ('0.08017768812741913',) We are sending mail with results at report@fotonower.com args[1336794898] : ('0.08017768812741913',) no score found for photo 1336794898 apple ('0.08017768812741913',) We are sending mail with results at report@fotonower.com args[1336794754] : ('0.08017768812741913',) no score found for photo 1336794754 apple ('0.08017768812741913',) We are sending mail with results at report@fotonower.com args[1336794688] : ('0.08017768812741913',) no score found for photo 1336794688 apple ('0.08017768812741913',) We are sending mail with results at report@fotonower.com refus_total : 0.08017768812741913 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20461697 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1336797871,1336793226,1336793329,1336793423,1336793492,1336793567,1336794688,1336794754,1336796211,1336794898,1336796103,1336797885,1336792620,1336797806,1336797703,1336794936,1336794462,1336793112,1336792626,1336792623) Found this number of photos: 20 begin to download photo : 1336797871 begin to download photo : 1336793567 begin to download photo : 1336796103 begin to download photo : 1336794936 download finish for photo 1336794936 begin to download photo : 1336794462 download finish for photo 1336793567 begin to download photo : 1336794688 download finish for photo 1336797871 begin to download photo : 1336793226 download finish for photo 1336794462 begin to download photo : 1336793112 download finish for photo 1336794688 begin to download photo : 1336794754 download finish for photo 1336793226 begin to download photo : 1336793329 download finish for photo 1336796103 begin to download photo : 1336797885 download finish for photo 1336793112 begin to download photo : 1336792626 download finish for photo 1336794754 begin to download photo : 1336796211 download finish for photo 1336793329 begin to download photo : 1336793423 download finish for photo 1336797885 begin to download photo : 1336792620 download finish for photo 1336796211 begin to download photo : 1336794898 download finish for photo 1336792626 begin to download photo : 1336792623 download finish for photo 1336793423 begin to download photo : 1336793492 download finish for photo 1336792620 begin to download photo : 1336797806 download finish for photo 1336793492 download finish for photo 1336792623 download finish for photo 1336794898 download finish for photo 1336797806 begin to download photo : 1336797703 download finish for photo 1336797703 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461697_11-02-2025_21_21_00.pdf results_Auto_P20461697_11-02-2025_21_21_00.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461697_11-02-2025_21_21_00.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20461697','results_Auto_P20461697_11-02-2025_21_21_00.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461697_11-02-2025_21_21_00.pdf','pdf','','0.78','0.08017768812741913') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1336797885, 1336797871, 1336797806, 1336797703, 1336796211, 1336796103, 1336794936, 1336794898, 1336794754, 1336794688] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797885', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797871', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797806', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797703', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796211', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796103', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794936', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794898', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794754', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794688', None, None, None, None, None, '2577749') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.014551401138305664 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.039272308349609 time spend to save output : 0.01485300064086914 total time spend for step 10 : 6.0541253089904785 step11:split_time_score Tue Feb 11 21:21:06 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('11', 20),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 11022025 20461697 Nombre de photos uploadées : 20 / 23040 (0%) 11022025 20461697 Nombre de photos taguées (types de déchets): 0 / 20 (0%) 11022025 20461697 Nombre de photos taguées (volume) : 0 / 20 (0%) elapsed_time : load_data_split_time_score 2.1457672119140625e-06 elapsed_time : order_list_meta_photo_and_scores 5.7220458984375e-06 ???????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.001203775405883789 elapsed_time : insert_dashboard_record_day_entry 0.024556636810302734 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461688 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461689 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461690 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461691 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461692 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461693 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461694 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461695 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461696 order by id desc limit 1 Qualite : 0.08017768812741913 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461697_11-02-2025_21_21_00.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461697 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461697 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461698 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461699 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461700 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461701 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461702 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461703 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461704 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461705 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461706 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461707 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461708 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461709 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461710 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461711 order by id desc limit 1 Qualite : 0.11839248806077622 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461712_11-02-2025_21_18_30.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461712 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461712 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461713 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461714 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461715 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461716 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461717 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461718 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461719 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'11022025': {'nb_upload': 20, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1336797885, 1336797871, 1336797806, 1336797703, 1336796211, 1336796103, 1336794936, 1336794898, 1336794754, 1336794688] Looping around the photos to save general results len do output : 1 /20461697Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797885', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797871', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797806', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336797703', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796211', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336796103', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794936', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794898', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794754', None, None, None, None, None, '2577749') ('3459', None, None, None, None, None, None, None, '2577749') ('3459', '20461697', '1336794688', None, None, None, None, None, '2577749') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013988971710205078 save_final save missing photos in datou_result : time spend for datou_step_exec : 23.681421041488647 time spend to save output : 0.014242887496948242 total time spend for step 11 : 23.695663928985596 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 156.73user 109.06system 5:04.47elapsed 87%CPU (0avgtext+0avgdata 3648044maxresident)k 3329744inputs+136376outputs (18017major+13291531minor)pagefaults 0swaps