python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1906744 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 21>10 please execute split_portfolio.py -i 20835307 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2615673'] with mtr_portfolio_ids : ['20835307'] and first list_photo_ids : [] new path : /proc/1906744/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.4799845218658447 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Wed Feb 26 11:38:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3896 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-26 11:38:33.824473: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-26 11:38:33.863273: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-26 11:38:33.865861: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fbdf4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-26 11:38:33.865908: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-26 11:38:33.871820: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-26 11:38:34.115230: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x20fae0c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-26 11:38:34.115277: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-26 11:38:34.116820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-26 11:38:34.117782: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-26 11:38:34.123937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-26 11:38:34.142194: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-26 11:38:34.143882: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-26 11:38:34.176232: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-26 11:38:34.180967: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-26 11:38:34.235946: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-26 11:38:34.237206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-26 11:38:34.237616: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-26 11:38:34.238290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-26 11:38:34.238308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-26 11:38:34.238318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-26 11:38:34.239798: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3444 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-26 11:38:34.574083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-26 11:38:34.574200: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-26 11:38:34.574227: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-26 11:38:34.574250: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-26 11:38:34.574274: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-26 11:38:34.574296: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-26 11:38:34.574319: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-26 11:38:34.574342: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-26 11:38:34.575498: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-26 11:38:34.576651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-26 11:38:34.576706: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-26 11:38:34.576728: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-26 11:38:34.576748: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-26 11:38:34.576768: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-26 11:38:34.576787: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-26 11:38:34.576807: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-26 11:38:34.576827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-26 11:38:34.577963: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-26 11:38:34.577997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-26 11:38:34.578008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-26 11:38:34.578018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-26 11:38:34.579236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3444 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-26 11:38:47.713287: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-26 11:38:47.902224: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-26 11:38:50.191892: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:50.978495: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-02-26 11:38:51.037315: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.037421: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.67GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 11:38:51.038196: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.038218: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.67GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 11:38:51.045160: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 11:38:51.045190: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 11:38:51.051497: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.051531: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 11:38:51.052228: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.052246: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 11:38:51.112552: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.112632: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 11:38:51.113299: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.113315: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 11:38:51.113967: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.113982: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 11:38:51.114620: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.114635: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 11:38:51.122505: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.123195: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.146745: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.147704: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.148608: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.149486: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.153944: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.154839: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.155767: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.156463: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.158322: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.158344: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-02-26 11:38:51.167775: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.168471: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.177671: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.178364: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.179039: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.179779: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.180478: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 11:38:51.181135: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.86G (3076390912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.83491 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 21 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 9 Detection mask done ! Trying to reset tf kernel 1907351 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2703 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3896 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0017681121826171875 nb_pixel_total : 47005 time to create 1 rle with old method : 0.06040143966674805 length of segment : 300 time for calcul the mask position with numpy : 0.43222713470458984 nb_pixel_total : 1278944 time to create 1 rle with new method : 0.26834774017333984 length of segment : 2623 time for calcul the mask position with numpy : 0.0009267330169677734 nb_pixel_total : 64591 time to create 1 rle with old method : 0.09130287170410156 length of segment : 267 time for calcul the mask position with numpy : 0.0011010169982910156 nb_pixel_total : 59399 time to create 1 rle with old method : 0.08520007133483887 length of segment : 296 time for calcul the mask position with numpy : 0.0009160041809082031 nb_pixel_total : 37884 time to create 1 rle with old method : 0.060320377349853516 length of segment : 217 time for calcul the mask position with numpy : 0.0057621002197265625 nb_pixel_total : 259683 time to create 1 rle with new method : 0.011337518692016602 length of segment : 871 time for calcul the mask position with numpy : 0.00015735626220703125 nb_pixel_total : 2763 time to create 1 rle with old method : 0.003389596939086914 length of segment : 97 time for calcul the mask position with numpy : 8.749961853027344e-05 nb_pixel_total : 2644 time to create 1 rle with old method : 0.003455638885498047 length of segment : 62 time for calcul the mask position with numpy : 0.0006308555603027344 nb_pixel_total : 19988 time to create 1 rle with old method : 0.02471017837524414 length of segment : 258 time for calcul the mask position with numpy : 0.0005228519439697266 nb_pixel_total : 25229 time to create 1 rle with old method : 0.030068397521972656 length of segment : 253 time for calcul the mask position with numpy : 0.00016617774963378906 nb_pixel_total : 2711 time to create 1 rle with old method : 0.003526926040649414 length of segment : 130 time for calcul the mask position with numpy : 0.00023674964904785156 nb_pixel_total : 10732 time to create 1 rle with old method : 0.012867212295532227 length of segment : 166 time for calcul the mask position with numpy : 0.0029044151306152344 nb_pixel_total : 171949 time to create 1 rle with new method : 0.008532285690307617 length of segment : 698 time for calcul the mask position with numpy : 0.0003476142883300781 nb_pixel_total : 15404 time to create 1 rle with old method : 0.018954992294311523 length of segment : 187 time for calcul the mask position with numpy : 0.0007920265197753906 nb_pixel_total : 49721 time to create 1 rle with old method : 0.060395240783691406 length of segment : 208 time for calcul the mask position with numpy : 0.0022873878479003906 nb_pixel_total : 151184 time to create 1 rle with new method : 0.004800319671630859 length of segment : 569 time for calcul the mask position with numpy : 0.0016093254089355469 nb_pixel_total : 103721 time to create 1 rle with old method : 0.12371134757995605 length of segment : 440 time for calcul the mask position with numpy : 0.20350122451782227 nb_pixel_total : 3817405 time to create 1 rle with new method : 0.7270264625549316 length of segment : 2492 time for calcul the mask position with numpy : 0.0004115104675292969 nb_pixel_total : 7433 time to create 1 rle with old method : 0.008962392807006836 length of segment : 144 time for calcul the mask position with numpy : 0.0003371238708496094 nb_pixel_total : 10213 time to create 1 rle with old method : 0.012745857238769531 length of segment : 106 time for calcul the mask position with numpy : 0.05235862731933594 nb_pixel_total : 1440301 time to create 1 rle with new method : 0.22326064109802246 length of segment : 2324 time for calcul the mask position with numpy : 0.004936933517456055 nb_pixel_total : 245334 time to create 1 rle with new method : 0.012616872787475586 length of segment : 1113 time for calcul the mask position with numpy : 0.006789445877075195 nb_pixel_total : 429315 time to create 1 rle with new method : 0.09090852737426758 length of segment : 728 time for calcul the mask position with numpy : 0.0009136199951171875 nb_pixel_total : 63730 time to create 1 rle with old method : 0.07384490966796875 length of segment : 531 time for calcul the mask position with numpy : 0.0036084651947021484 nb_pixel_total : 105999 time to create 1 rle with old method : 0.12709546089172363 length of segment : 389 time for calcul the mask position with numpy : 0.000990152359008789 nb_pixel_total : 32605 time to create 1 rle with old method : 0.037813663482666016 length of segment : 228 time for calcul the mask position with numpy : 0.00032329559326171875 nb_pixel_total : 7816 time to create 1 rle with old method : 0.009689569473266602 length of segment : 96 time for calcul the mask position with numpy : 0.00015616416931152344 nb_pixel_total : 2543 time to create 1 rle with old method : 0.003323078155517578 length of segment : 57 time for calcul the mask position with numpy : 0.0015666484832763672 nb_pixel_total : 55126 time to create 1 rle with old method : 0.06711030006408691 length of segment : 321 time for calcul the mask position with numpy : 0.0008604526519775391 nb_pixel_total : 19911 time to create 1 rle with old method : 0.02396368980407715 length of segment : 174 time for calcul the mask position with numpy : 0.0006690025329589844 nb_pixel_total : 15572 time to create 1 rle with old method : 0.018785476684570312 length of segment : 294 time for calcul the mask position with numpy : 0.0004680156707763672 nb_pixel_total : 15889 time to create 1 rle with old method : 0.01906418800354004 length of segment : 240 time for calcul the mask position with numpy : 0.0008153915405273438 nb_pixel_total : 22455 time to create 1 rle with old method : 0.026739835739135742 length of segment : 194 time for calcul the mask position with numpy : 0.00556182861328125 nb_pixel_total : 237146 time to create 1 rle with new method : 0.009673357009887695 length of segment : 787 time for calcul the mask position with numpy : 0.0002117156982421875 nb_pixel_total : 2828 time to create 1 rle with old method : 0.0034856796264648438 length of segment : 98 time for calcul the mask position with numpy : 0.0020606517791748047 nb_pixel_total : 41829 time to create 1 rle with old method : 0.047530412673950195 length of segment : 803 time for calcul the mask position with numpy : 0.0040743350982666016 nb_pixel_total : 182086 time to create 1 rle with new method : 0.006923198699951172 length of segment : 587 time for calcul the mask position with numpy : 0.0011985301971435547 nb_pixel_total : 30490 time to create 1 rle with old method : 0.03604936599731445 length of segment : 349 time for calcul the mask position with numpy : 0.0036504268646240234 nb_pixel_total : 183244 time to create 1 rle with new method : 0.01021265983581543 length of segment : 607 time for calcul the mask position with numpy : 0.00033855438232421875 nb_pixel_total : 9983 time to create 1 rle with old method : 0.01159358024597168 length of segment : 125 time for calcul the mask position with numpy : 0.0001251697540283203 nb_pixel_total : 1776 time to create 1 rle with old method : 0.0022721290588378906 length of segment : 55 time for calcul the mask position with numpy : 0.0047245025634765625 nb_pixel_total : 180853 time to create 1 rle with new method : 0.0073337554931640625 length of segment : 462 time for calcul the mask position with numpy : 0.0006544589996337891 nb_pixel_total : 16664 time to create 1 rle with old method : 0.020684480667114258 length of segment : 155 time for calcul the mask position with numpy : 0.0007686614990234375 nb_pixel_total : 26343 time to create 1 rle with old method : 0.03193187713623047 length of segment : 182 time for calcul the mask position with numpy : 0.0020194053649902344 nb_pixel_total : 87410 time to create 1 rle with old method : 0.10005760192871094 length of segment : 366 time for calcul the mask position with numpy : 0.00035190582275390625 nb_pixel_total : 7051 time to create 1 rle with old method : 0.00860738754272461 length of segment : 92 time for calcul the mask position with numpy : 0.0005183219909667969 nb_pixel_total : 10568 time to create 1 rle with old method : 0.012555599212646484 length of segment : 140 time for calcul the mask position with numpy : 0.0003197193145751953 nb_pixel_total : 5254 time to create 1 rle with old method : 0.006716251373291016 length of segment : 108 time for calcul the mask position with numpy : 0.0003380775451660156 nb_pixel_total : 8384 time to create 1 rle with old method : 0.01003718376159668 length of segment : 122 time for calcul the mask position with numpy : 0.0005822181701660156 nb_pixel_total : 16633 time to create 1 rle with old method : 0.01937723159790039 length of segment : 147 time for calcul the mask position with numpy : 0.0019769668579101562 nb_pixel_total : 81707 time to create 1 rle with old method : 0.09416055679321289 length of segment : 395 time for calcul the mask position with numpy : 0.0014765262603759766 nb_pixel_total : 57062 time to create 1 rle with old method : 0.06672978401184082 length of segment : 345 time for calcul the mask position with numpy : 0.0001552104949951172 nb_pixel_total : 1056 time to create 1 rle with old method : 0.0014214515686035156 length of segment : 44 time for calcul the mask position with numpy : 0.0030286312103271484 nb_pixel_total : 88146 time to create 1 rle with old method : 0.10039496421813965 length of segment : 459 time for calcul the mask position with numpy : 0.0005104541778564453 nb_pixel_total : 8611 time to create 1 rle with old method : 0.010622739791870117 length of segment : 167 time for calcul the mask position with numpy : 0.0008189678192138672 nb_pixel_total : 28816 time to create 1 rle with old method : 0.03429722785949707 length of segment : 147 time for calcul the mask position with numpy : 0.0019381046295166016 nb_pixel_total : 40667 time to create 1 rle with old method : 0.0476839542388916 length of segment : 372 time for calcul the mask position with numpy : 0.003073453903198242 nb_pixel_total : 132606 time to create 1 rle with old method : 0.149094820022583 length of segment : 623 time for calcul the mask position with numpy : 0.07750892639160156 nb_pixel_total : 2049979 time to create 1 rle with new method : 0.3082008361816406 length of segment : 2083 time for calcul the mask position with numpy : 0.0009427070617675781 nb_pixel_total : 42577 time to create 1 rle with old method : 0.049074411392211914 length of segment : 185 time for calcul the mask position with numpy : 0.0015301704406738281 nb_pixel_total : 101375 time to create 1 rle with old method : 0.11597561836242676 length of segment : 416 time for calcul the mask position with numpy : 0.0037081241607666016 nb_pixel_total : 203916 time to create 1 rle with new method : 0.008649349212646484 length of segment : 474 time for calcul the mask position with numpy : 0.0017743110656738281 nb_pixel_total : 142413 time to create 1 rle with old method : 0.1883087158203125 length of segment : 412 time for calcul the mask position with numpy : 0.0026357173919677734 nb_pixel_total : 138658 time to create 1 rle with old method : 0.1727447509765625 length of segment : 816 time for calcul the mask position with numpy : 0.012712478637695312 nb_pixel_total : 941233 time to create 1 rle with new method : 0.0685727596282959 length of segment : 1424 time for calcul the mask position with numpy : 0.00010919570922851562 nb_pixel_total : 1664 time to create 1 rle with old method : 0.002053976058959961 length of segment : 84 time for calcul the mask position with numpy : 0.0004131793975830078 nb_pixel_total : 12599 time to create 1 rle with old method : 0.015226125717163086 length of segment : 120 time for calcul the mask position with numpy : 0.0013873577117919922 nb_pixel_total : 41917 time to create 1 rle with old method : 0.051509857177734375 length of segment : 212 time for calcul the mask position with numpy : 0.004261016845703125 nb_pixel_total : 100506 time to create 1 rle with old method : 0.12023019790649414 length of segment : 357 time for calcul the mask position with numpy : 0.0008108615875244141 nb_pixel_total : 31754 time to create 1 rle with old method : 0.037334442138671875 length of segment : 265 time for calcul the mask position with numpy : 0.0002200603485107422 nb_pixel_total : 3584 time to create 1 rle with old method : 0.004953145980834961 length of segment : 58 time for calcul the mask position with numpy : 0.0026848316192626953 nb_pixel_total : 73410 time to create 1 rle with old method : 0.10382080078125 length of segment : 677 time for calcul the mask position with numpy : 0.0071256160736083984 nb_pixel_total : 338125 time to create 1 rle with new method : 0.012499094009399414 length of segment : 578 time for calcul the mask position with numpy : 0.0017354488372802734 nb_pixel_total : 56773 time to create 1 rle with old method : 0.0733652114868164 length of segment : 291 time for calcul the mask position with numpy : 0.001989603042602539 nb_pixel_total : 79974 time to create 1 rle with old method : 0.1072850227355957 length of segment : 405 time for calcul the mask position with numpy : 0.0003459453582763672 nb_pixel_total : 2901 time to create 1 rle with old method : 0.004660844802856445 length of segment : 103 time for calcul the mask position with numpy : 0.0017821788787841797 nb_pixel_total : 53451 time to create 1 rle with old method : 0.06903433799743652 length of segment : 311 time for calcul the mask position with numpy : 0.005323648452758789 nb_pixel_total : 321561 time to create 1 rle with new method : 0.011127233505249023 length of segment : 448 time for calcul the mask position with numpy : 0.0020589828491210938 nb_pixel_total : 119191 time to create 1 rle with old method : 0.15148448944091797 length of segment : 416 time for calcul the mask position with numpy : 0.006216764450073242 nb_pixel_total : 169132 time to create 1 rle with new method : 0.007447481155395508 length of segment : 572 time for calcul the mask position with numpy : 0.000217437744140625 nb_pixel_total : 2552 time to create 1 rle with old method : 0.0032660961151123047 length of segment : 81 time for calcul the mask position with numpy : 0.005759716033935547 nb_pixel_total : 188409 time to create 1 rle with new method : 0.011674642562866211 length of segment : 558 time for calcul the mask position with numpy : 0.0006577968597412109 nb_pixel_total : 19378 time to create 1 rle with old method : 0.0234830379486084 length of segment : 174 time for calcul the mask position with numpy : 0.002519845962524414 nb_pixel_total : 112368 time to create 1 rle with old method : 0.14196085929870605 length of segment : 525 time for calcul the mask position with numpy : 0.00033974647521972656 nb_pixel_total : 2686 time to create 1 rle with old method : 0.004431486129760742 length of segment : 94 time for calcul the mask position with numpy : 0.0007216930389404297 nb_pixel_total : 25276 time to create 1 rle with old method : 0.03306460380554199 length of segment : 145 time for calcul the mask position with numpy : 0.0049953460693359375 nb_pixel_total : 195953 time to create 1 rle with new method : 0.009400606155395508 length of segment : 487 time for calcul the mask position with numpy : 0.0021448135375976562 nb_pixel_total : 59956 time to create 1 rle with old method : 0.06856942176818848 length of segment : 408 time for calcul the mask position with numpy : 0.0019409656524658203 nb_pixel_total : 66140 time to create 1 rle with old method : 0.08113312721252441 length of segment : 340 time for calcul the mask position with numpy : 0.0002980232238769531 nb_pixel_total : 7389 time to create 1 rle with old method : 0.009112358093261719 length of segment : 111 time for calcul the mask position with numpy : 0.0007491111755371094 nb_pixel_total : 26728 time to create 1 rle with old method : 0.038800716400146484 length of segment : 202 time for calcul the mask position with numpy : 0.00504302978515625 nb_pixel_total : 226245 time to create 1 rle with new method : 0.01220703125 length of segment : 496 time for calcul the mask position with numpy : 0.0024538040161132812 nb_pixel_total : 122700 time to create 1 rle with old method : 0.14405083656311035 length of segment : 332 time for calcul the mask position with numpy : 0.0005054473876953125 nb_pixel_total : 12598 time to create 1 rle with old method : 0.015436887741088867 length of segment : 103 time for calcul the mask position with numpy : 0.0001652240753173828 nb_pixel_total : 2664 time to create 1 rle with old method : 0.0032889842987060547 length of segment : 80 time for calcul the mask position with numpy : 0.002672910690307617 nb_pixel_total : 87152 time to create 1 rle with old method : 0.10936307907104492 length of segment : 557 time for calcul the mask position with numpy : 0.006346225738525391 nb_pixel_total : 246069 time to create 1 rle with new method : 0.012972593307495117 length of segment : 631 time for calcul the mask position with numpy : 0.00099945068359375 nb_pixel_total : 27743 time to create 1 rle with old method : 0.03305411338806152 length of segment : 309 time for calcul the mask position with numpy : 0.0034127235412597656 nb_pixel_total : 126140 time to create 1 rle with old method : 0.14682841300964355 length of segment : 430 time for calcul the mask position with numpy : 0.0007195472717285156 nb_pixel_total : 9047 time to create 1 rle with old method : 0.01094365119934082 length of segment : 148 time for calcul the mask position with numpy : 0.0021791458129882812 nb_pixel_total : 59900 time to create 1 rle with old method : 0.09399867057800293 length of segment : 413 time for calcul the mask position with numpy : 0.0068073272705078125 nb_pixel_total : 297502 time to create 1 rle with new method : 0.015333175659179688 length of segment : 662 time for calcul the mask position with numpy : 0.0014667510986328125 nb_pixel_total : 50547 time to create 1 rle with old method : 0.07052826881408691 length of segment : 246 time for calcul the mask position with numpy : 0.000186920166015625 nb_pixel_total : 3082 time to create 1 rle with old method : 0.004836320877075195 length of segment : 48 time for calcul the mask position with numpy : 0.0001881122589111328 nb_pixel_total : 956 time to create 1 rle with old method : 0.001428842544555664 length of segment : 68 time for calcul the mask position with numpy : 0.004019498825073242 nb_pixel_total : 82322 time to create 1 rle with old method : 0.11249256134033203 length of segment : 583 time for calcul the mask position with numpy : 0.00042700767517089844 nb_pixel_total : 3469 time to create 1 rle with old method : 0.004128932952880859 length of segment : 138 time for calcul the mask position with numpy : 0.001596689224243164 nb_pixel_total : 18431 time to create 1 rle with old method : 0.022885799407958984 length of segment : 208 time for calcul the mask position with numpy : 0.00021839141845703125 nb_pixel_total : 7030 time to create 1 rle with old method : 0.008668184280395508 length of segment : 85 time for calcul the mask position with numpy : 0.00012540817260742188 nb_pixel_total : 2420 time to create 1 rle with old method : 0.002912282943725586 length of segment : 97 time for calcul the mask position with numpy : 0.0005476474761962891 nb_pixel_total : 10922 time to create 1 rle with old method : 0.013047218322753906 length of segment : 147 time for calcul the mask position with numpy : 0.0005674362182617188 nb_pixel_total : 15013 time to create 1 rle with old method : 0.019017934799194336 length of segment : 289 time for calcul the mask position with numpy : 0.23386740684509277 nb_pixel_total : 3524222 time to create 1 rle with new method : 0.5628688335418701 length of segment : 2083 time for calcul the mask position with numpy : 0.0015177726745605469 nb_pixel_total : 44143 time to create 1 rle with old method : 0.05930185317993164 length of segment : 359 time for calcul the mask position with numpy : 0.0011467933654785156 nb_pixel_total : 61750 time to create 1 rle with old method : 0.07235193252563477 length of segment : 389 time for calcul the mask position with numpy : 0.0006873607635498047 nb_pixel_total : 40408 time to create 1 rle with old method : 0.05306744575500488 length of segment : 476 time for calcul the mask position with numpy : 0.001249074935913086 nb_pixel_total : 72068 time to create 1 rle with old method : 0.08863091468811035 length of segment : 331 time for calcul the mask position with numpy : 0.00047206878662109375 nb_pixel_total : 28683 time to create 1 rle with old method : 0.03328847885131836 length of segment : 222 time spent for convertir_results : 17.49089813232422 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 118 chid ids of type : 3663 Number RLEs to save : 48572 save missing photos in datou_result : time spend for datou_step_exec : 76.78431510925293 time spend to save output : 3.255345344543457 total time spend for step 1 : 80.03966045379639 step2:crop_condition Wed Feb 26 11:39:49 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 118 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 41 About to insert : list_path_to_insert length 41 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1339695618, 1339695553, 1339695551, 1339695550, 1339695548, 1339695545, 1339695498, 1339695497, 1339695496, 1339695495] Looping around the photos to save general results len do output : 73 /-3687390920Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390919Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390929Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390941Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390944Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390966Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390983Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390987Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391005Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391029Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391031Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390930Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390948Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390965Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391007Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390931Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390954Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390960Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390997Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390994Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391008Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390924Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390918Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390916Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390921Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390932Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390928Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390939Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390953Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390952Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390947Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390945Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390949Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390969Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390971Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390972Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390967Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390959Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390962Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390976Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390975Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390977Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390978Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390995Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390993Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390990Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390989Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390999Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391004Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391006Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391002Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391012Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391011Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391013Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391003Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391023Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391021Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391017Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391026Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391032Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391030Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391033Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390942Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390961Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390964Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390992Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391018Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390925Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390935Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390973Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390979Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687391014Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687390998Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695618', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695553', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695551', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695550', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695548', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695545', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695498', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695497', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695496', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695495', None, None, None, None, None, '2615673') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 229 time used for this insertion : 0.031324148178100586 save_final save missing photos in datou_result : time spend for datou_step_exec : 35.95807647705078 time spend to save output : 0.033802032470703125 total time spend for step 2 : 35.991878509521484 step3:thcl Wed Feb 26 11:40:25 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.013098716735839844 time to convert the images to numpy array : 0.026495933532714844 time to import caffe and check if the image exist : 0.012026786804199219 time to convert the images to numpy array : 0.8818624019622803 time to import caffe and check if the image exist : 0.02020263671875 time to convert the images to numpy array : 1.3259716033935547 time to import caffe and check if the image exist : 0.021378755569458008 time to convert the images to numpy array : 1.3292179107666016 time to import caffe and check if the image exist : 0.6894028186798096 time to convert the images to numpy array : 0.6672835350036621 time to import caffe and check if the image exist : 0.02924060821533203 time to convert the images to numpy array : 1.3339176177978516 time to import caffe and check if the image exist : 0.016681432723999023 time to convert the images to numpy array : 1.3526873588562012 time to import caffe and check if the image exist : 0.01454305648803711 time to convert the images to numpy array : 1.3567986488342285 time to import caffe and check if the image exist : 0.013759851455688477 time to convert the images to numpy array : 1.3691370487213135 time to import caffe and check if the image exist : 0.02185201644897461 time to convert the images to numpy array : 1.3753232955932617 total time to convert the images to numpy array : 2.215724229812622 list photo_ids error: [] list photo_ids correct : [-3687390998, -3687390964, -3687390992, -3687391018, -3687390925, -3687390935, -3687390973, -3687390979, -3687391014, -3687391021, -3687391017, -3687391026, -3687391032, -3687391030, -3687391033, -3687390942, -3687390961, -3687391005, -3687391029, -3687391031, -3687390930, -3687390948, -3687390965, -3687391007, -3687390931, -3687390920, -3687390919, -3687390929, -3687390941, -3687390944, -3687390966, -3687390983, -3687390987, -3687390949, -3687390969, -3687390971, -3687390972, -3687390967, -3687390959, -3687390962, -3687390976, -3687391004, -3687391006, -3687391002, -3687391012, -3687391011, -3687391013, -3687391003, -3687391023, -3687390954, -3687390960, -3687390997, -3687390994, -3687391008, -3687390924, -3687390918, -3687390916, -3687390921, -3687390932, -3687390928, -3687390939, -3687390953, -3687390952, -3687390947, -3687390945, -3687390975, -3687390977, -3687390978, -3687390995, -3687390993, -3687390990, -3687390989, -3687390999] number of photos to traite : 73 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3896 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3677 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.8720169067382812 time used to do the prediction : 0.2037198543548584 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3675 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3436 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.0649213790893555 time used to do the prediction : 0.17438459396362305 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.384185791015625e-05 save missing photos in datou_result : time spend for datou_step_exec : 11.564496994018555 time spend to save output : 0.000553131103515625 total time spend for step 3 : 11.56505012512207 step4:merge_mask_thcl_custom Wed Feb 26 11:40:37 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 118 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1339695618, 1339695553, 1339695551, 1339695550, 1339695548, 1339695545, 1339695498, 1339695497, 1339695496, 1339695495] Looping around the photos to save general results len do output : 10 /1339695618Didn't retrieve data .Didn't retrieve data . /1339695553Didn't retrieve data .Didn't retrieve data . /1339695551Didn't retrieve data .Didn't retrieve data . /1339695550Didn't retrieve data .Didn't retrieve data . /1339695548Didn't retrieve data .Didn't retrieve data . /1339695545Didn't retrieve data .Didn't retrieve data . /1339695498Didn't retrieve data .Didn't retrieve data . /1339695497Didn't retrieve data .Didn't retrieve data . /1339695496Didn't retrieve data .Didn't retrieve data . /1339695495Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695618', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695553', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695551', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695550', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695548', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695545', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695498', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695497', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695496', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695495', None, None, None, None, None, '2615673') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014905929565429688 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11707305908203125 time spend to save output : 0.015508174896240234 total time spend for step 4 : 0.13258123397827148 step5:rle_unique_nms_with_priority Wed Feb 26 11:40:37 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 10 nb_hashtags : 6 time to prepare the origin masks : 10.255561351776123 time for calcul the mask position with numpy : 1.1056900024414062 nb_pixel_total : 5285279 time to create 1 rle with new method : 0.6773045063018799 time for calcul the mask position with numpy : 0.028005599975585938 nb_pixel_total : 259278 time to create 1 rle with new method : 0.6684393882751465 time for calcul the mask position with numpy : 0.029482364654541016 nb_pixel_total : 46672 time to create 1 rle with old method : 0.05764889717102051 time for calcul the mask position with numpy : 0.0246584415435791 nb_pixel_total : 37757 time to create 1 rle with old method : 0.043212175369262695 time for calcul the mask position with numpy : 0.02250051498413086 nb_pixel_total : 5350 time to create 1 rle with old method : 0.006601572036743164 time for calcul the mask position with numpy : 0.022857189178466797 nb_pixel_total : 59626 time to create 1 rle with old method : 0.06604957580566406 time for calcul the mask position with numpy : 0.02282238006591797 nb_pixel_total : 2638 time to create 1 rle with old method : 0.0029892921447753906 time for calcul the mask position with numpy : 0.03709101676940918 nb_pixel_total : 2757 time to create 1 rle with old method : 0.0031414031982421875 time for calcul the mask position with numpy : 0.03643321990966797 nb_pixel_total : 815 time to create 1 rle with old method : 0.0009853839874267578 time for calcul the mask position with numpy : 0.07644534111022949 nb_pixel_total : 1330148 time to create 1 rle with new method : 0.5875353813171387 time for calcul the mask position with numpy : 0.0387568473815918 nb_pixel_total : 19920 time to create 1 rle with old method : 0.022603273391723633 create new chi : 3.672030448913574 time to delete rle : 0.013866901397705078 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 11213 TO DO : save crop sub photo not yet done ! save time : 0.6930418014526367 nb_obj : 7 nb_hashtags : 5 time to prepare the origin masks : 7.386603116989136 time for calcul the mask position with numpy : 0.5864224433898926 nb_pixel_total : 6545639 time to create 1 rle with new method : 0.6188061237335205 time for calcul the mask position with numpy : 0.03415060043334961 nb_pixel_total : 103663 time to create 1 rle with old method : 0.12121129035949707 time for calcul the mask position with numpy : 0.05116701126098633 nb_pixel_total : 49448 time to create 1 rle with old method : 0.07936859130859375 time for calcul the mask position with numpy : 0.08986377716064453 nb_pixel_total : 151231 time to create 1 rle with new method : 0.7254853248596191 time for calcul the mask position with numpy : 0.04166388511657715 nb_pixel_total : 15357 time to create 1 rle with old method : 0.017398834228515625 time for calcul the mask position with numpy : 0.039825439453125 nb_pixel_total : 10648 time to create 1 rle with old method : 0.011745214462280273 time for calcul the mask position with numpy : 0.03949689865112305 nb_pixel_total : 2697 time to create 1 rle with old method : 0.0032041072845458984 time for calcul the mask position with numpy : 0.03470039367675781 nb_pixel_total : 171557 time to create 1 rle with new method : 0.5747926235198975 create new chi : 3.1804747581481934 time to delete rle : 0.0013620853424072266 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6980 TO DO : save crop sub photo not yet done ! save time : 0.5168895721435547 nb_obj : 6 nb_hashtags : 2 time to prepare the origin masks : 7.730425596237183 time for calcul the mask position with numpy : 0.16471600532531738 nb_pixel_total : 2870501 time to create 1 rle with new method : 1.2826159000396729 time for calcul the mask position with numpy : 0.04010152816772461 nb_pixel_total : 62802 time to create 1 rle with old method : 0.07077670097351074 time for calcul the mask position with numpy : 0.03618621826171875 nb_pixel_total : 169599 time to create 1 rle with new method : 1.0523560047149658 time for calcul the mask position with numpy : 0.03268170356750488 nb_pixel_total : 3201 time to create 1 rle with old method : 0.0038766860961914062 time for calcul the mask position with numpy : 0.03412365913391113 nb_pixel_total : 7406 time to create 1 rle with old method : 0.009113788604736328 time for calcul the mask position with numpy : 0.3878743648529053 nb_pixel_total : 3873133 time to create 1 rle with new method : 0.4160928726196289 time for calcul the mask position with numpy : 0.030266284942626953 nb_pixel_total : 63598 time to create 1 rle with old method : 0.07370996475219727 create new chi : 3.736384153366089 time to delete rle : 0.0011582374572753906 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 11326 TO DO : save crop sub photo not yet done ! save time : 0.8934957981109619 nb_obj : 15 nb_hashtags : 6 time to prepare the origin masks : 10.759005069732666 time for calcul the mask position with numpy : 0.37557387351989746 nb_pixel_total : 6277395 time to create 1 rle with new method : 0.4932856559753418 time for calcul the mask position with numpy : 0.03520917892456055 nb_pixel_total : 30367 time to create 1 rle with old method : 0.03404974937438965 time for calcul the mask position with numpy : 0.043399810791015625 nb_pixel_total : 22299 time to create 1 rle with old method : 0.04033327102661133 time for calcul the mask position with numpy : 0.042725563049316406 nb_pixel_total : 41737 time to create 1 rle with old method : 0.04829978942871094 time for calcul the mask position with numpy : 0.04106283187866211 nb_pixel_total : 2821 time to create 1 rle with old method : 0.0033028125762939453 time for calcul the mask position with numpy : 0.04349350929260254 nb_pixel_total : 18081 time to create 1 rle with old method : 0.022162437438964844 time for calcul the mask position with numpy : 0.046207427978515625 nb_pixel_total : 7769 time to create 1 rle with old method : 0.00964975357055664 time for calcul the mask position with numpy : 0.03953719139099121 nb_pixel_total : 55039 time to create 1 rle with old method : 0.06255412101745605 time for calcul the mask position with numpy : 0.03991293907165527 nb_pixel_total : 15515 time to create 1 rle with old method : 0.02453303337097168 time for calcul the mask position with numpy : 0.05939507484436035 nb_pixel_total : 32525 time to create 1 rle with old method : 0.05391097068786621 time for calcul the mask position with numpy : 0.0450587272644043 nb_pixel_total : 15836 time to create 1 rle with old method : 0.017848730087280273 time for calcul the mask position with numpy : 0.035919904708862305 nb_pixel_total : 2517 time to create 1 rle with old method : 0.003141641616821289 time for calcul the mask position with numpy : 0.04644346237182617 nb_pixel_total : 236888 time to create 1 rle with new method : 0.6443924903869629 time for calcul the mask position with numpy : 0.04275202751159668 nb_pixel_total : 105442 time to create 1 rle with old method : 0.13042402267456055 time for calcul the mask position with numpy : 0.03458070755004883 nb_pixel_total : 4072 time to create 1 rle with old method : 0.02413034439086914 time for calcul the mask position with numpy : 0.03975939750671387 nb_pixel_total : 181937 time to create 1 rle with new method : 0.7032458782196045 create new chi : 3.4278335571289062 time to delete rle : 0.0019714832305908203 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 11927 TO DO : save crop sub photo not yet done ! save time : 0.793447732925415 nb_obj : 18 nb_hashtags : 7 time to prepare the origin masks : 9.608170509338379 time for calcul the mask position with numpy : 0.8208637237548828 nb_pixel_total : 6252481 time to create 1 rle with new method : 0.8559560775756836 time for calcul the mask position with numpy : 0.06435012817382812 nb_pixel_total : 8350 time to create 1 rle with old method : 0.011708974838256836 time for calcul the mask position with numpy : 0.05439352989196777 nb_pixel_total : 81593 time to create 1 rle with old method : 0.09541463851928711 time for calcul the mask position with numpy : 0.057442665100097656 nb_pixel_total : 7009 time to create 1 rle with old method : 0.00807952880859375 time for calcul the mask position with numpy : 0.053877830505371094 nb_pixel_total : 28638 time to create 1 rle with old method : 0.03338146209716797 time for calcul the mask position with numpy : 0.0611271858215332 nb_pixel_total : 87823 time to create 1 rle with old method : 0.09957695007324219 time for calcul the mask position with numpy : 0.05532956123352051 nb_pixel_total : 8574 time to create 1 rle with old method : 0.012105941772460938 time for calcul the mask position with numpy : 0.06048464775085449 nb_pixel_total : 26256 time to create 1 rle with old method : 0.03090524673461914 time for calcul the mask position with numpy : 0.05541729927062988 nb_pixel_total : 16511 time to create 1 rle with old method : 0.01987481117248535 time for calcul the mask position with numpy : 0.05592799186706543 nb_pixel_total : 132391 time to create 1 rle with old method : 0.18446898460388184 time for calcul the mask position with numpy : 0.0555875301361084 nb_pixel_total : 10520 time to create 1 rle with old method : 0.018037796020507812 time for calcul the mask position with numpy : 0.061466217041015625 nb_pixel_total : 1041 time to create 1 rle with old method : 0.0018639564514160156 time for calcul the mask position with numpy : 0.05876445770263672 nb_pixel_total : 5226 time to create 1 rle with old method : 0.007550716400146484 time for calcul the mask position with numpy : 0.05095314979553223 nb_pixel_total : 87275 time to create 1 rle with old method : 0.10032248497009277 time for calcul the mask position with numpy : 0.04317927360534668 nb_pixel_total : 57034 time to create 1 rle with old method : 0.06670808792114258 time for calcul the mask position with numpy : 0.04350423812866211 nb_pixel_total : 16644 time to create 1 rle with old method : 0.0187835693359375 time for calcul the mask position with numpy : 0.047456979751586914 nb_pixel_total : 180536 time to create 1 rle with new method : 0.9375705718994141 time for calcul the mask position with numpy : 0.04247236251831055 nb_pixel_total : 1770 time to create 1 rle with old method : 0.0021085739135742188 time for calcul the mask position with numpy : 0.04745316505432129 nb_pixel_total : 40568 time to create 1 rle with old method : 0.04751181602478027 create new chi : 4.438684701919556 time to delete rle : 0.0019421577453613281 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 10980 TO DO : save crop sub photo not yet done ! save time : 3.3197476863861084 nb_obj : 8 nb_hashtags : 3 time to prepare the origin masks : 6.175699710845947 time for calcul the mask position with numpy : 0.26163554191589355 nb_pixel_total : 4257797 time to create 1 rle with new method : 0.4998815059661865 time for calcul the mask position with numpy : 0.030333280563354492 nb_pixel_total : 101329 time to create 1 rle with old method : 0.14478206634521484 time for calcul the mask position with numpy : 0.031210660934448242 nb_pixel_total : 1655 time to create 1 rle with old method : 0.0019283294677734375 time for calcul the mask position with numpy : 0.034720659255981445 nb_pixel_total : 114895 time to create 1 rle with old method : 0.12872076034545898 time for calcul the mask position with numpy : 0.031197547912597656 nb_pixel_total : 202806 time to create 1 rle with new method : 1.4001827239990234 time for calcul the mask position with numpy : 0.08266639709472656 nb_pixel_total : 2048700 time to create 1 rle with new method : 0.70229172706604 time for calcul the mask position with numpy : 0.04636979103088379 nb_pixel_total : 142066 time to create 1 rle with old method : 0.18425869941711426 time for calcul the mask position with numpy : 0.03558611869812012 nb_pixel_total : 138562 time to create 1 rle with old method : 0.16249537467956543 time for calcul the mask position with numpy : 0.035002946853637695 nb_pixel_total : 42430 time to create 1 rle with old method : 0.05099797248840332 create new chi : 3.991927146911621 time to delete rle : 0.0018601417541503906 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 12964 TO DO : save crop sub photo not yet done ! save time : 0.7498373985290527 nb_obj : 18 nb_hashtags : 7 time to prepare the origin masks : 11.483614683151245 time for calcul the mask position with numpy : 0.58176589012146 nb_pixel_total : 5800352 time to create 1 rle with new method : 0.7475693225860596 time for calcul the mask position with numpy : 0.03811311721801758 nb_pixel_total : 188119 time to create 1 rle with new method : 0.5648980140686035 time for calcul the mask position with numpy : 0.04180288314819336 nb_pixel_total : 79717 time to create 1 rle with old method : 0.09169721603393555 time for calcul the mask position with numpy : 0.0353398323059082 nb_pixel_total : 73417 time to create 1 rle with old method : 0.08711910247802734 time for calcul the mask position with numpy : 0.03734946250915527 nb_pixel_total : 41722 time to create 1 rle with old method : 0.047502756118774414 time for calcul the mask position with numpy : 0.03950667381286621 nb_pixel_total : 5180 time to create 1 rle with old method : 0.00891566276550293 time for calcul the mask position with numpy : 0.04422926902770996 nb_pixel_total : 54787 time to create 1 rle with old method : 0.0647430419921875 time for calcul the mask position with numpy : 0.036911964416503906 nb_pixel_total : 2546 time to create 1 rle with old method : 0.0029768943786621094 time for calcul the mask position with numpy : 0.03904986381530762 nb_pixel_total : 19313 time to create 1 rle with old method : 0.02209782600402832 time for calcul the mask position with numpy : 0.03583645820617676 nb_pixel_total : 2852 time to create 1 rle with old method : 0.003608226776123047 time for calcul the mask position with numpy : 0.03866004943847656 nb_pixel_total : 168667 time to create 1 rle with new method : 0.5695674419403076 time for calcul the mask position with numpy : 0.02783036231994629 nb_pixel_total : 7588 time to create 1 rle with old method : 0.009884119033813477 time for calcul the mask position with numpy : 0.029253721237182617 nb_pixel_total : 6529 time to create 1 rle with old method : 0.009967327117919922 time for calcul the mask position with numpy : 0.03378891944885254 nb_pixel_total : 337643 time to create 1 rle with new method : 0.3842942714691162 time for calcul the mask position with numpy : 0.024230241775512695 nb_pixel_total : 113558 time to create 1 rle with old method : 0.12811899185180664 time for calcul the mask position with numpy : 0.02296924591064453 nb_pixel_total : 3527 time to create 1 rle with old method : 0.0048291683197021484 time for calcul the mask position with numpy : 0.025242090225219727 nb_pixel_total : 31711 time to create 1 rle with old method : 0.03745579719543457 time for calcul the mask position with numpy : 0.02466106414794922 nb_pixel_total : 100451 time to create 1 rle with old method : 0.11300897598266602 time for calcul the mask position with numpy : 0.022986173629760742 nb_pixel_total : 12561 time to create 1 rle with old method : 0.014748811721801758 create new chi : 4.355132818222046 time to delete rle : 0.0022420883178710938 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 13069 TO DO : save crop sub photo not yet done ! save time : 0.7482490539550781 nb_obj : 14 nb_hashtags : 7 time to prepare the origin masks : 11.453305006027222 time for calcul the mask position with numpy : 0.4795494079589844 nb_pixel_total : 6157532 time to create 1 rle with new method : 0.5642135143280029 time for calcul the mask position with numpy : 0.030315876007080078 nb_pixel_total : 12111 time to create 1 rle with old method : 0.016011953353881836 time for calcul the mask position with numpy : 0.03275036811828613 nb_pixel_total : 122340 time to create 1 rle with old method : 0.13324522972106934 time for calcul the mask position with numpy : 0.033591270446777344 nb_pixel_total : 65970 time to create 1 rle with old method : 0.0733346939086914 time for calcul the mask position with numpy : 0.03860330581665039 nb_pixel_total : 195738 time to create 1 rle with new method : 0.5337259769439697 time for calcul the mask position with numpy : 0.039620161056518555 nb_pixel_total : 86718 time to create 1 rle with old method : 0.10271668434143066 time for calcul the mask position with numpy : 0.04055023193359375 nb_pixel_total : 245319 time to create 1 rle with new method : 0.8593811988830566 time for calcul the mask position with numpy : 0.031569480895996094 nb_pixel_total : 26602 time to create 1 rle with old method : 0.03204035758972168 time for calcul the mask position with numpy : 0.03171133995056152 nb_pixel_total : 27735 time to create 1 rle with old method : 0.030408143997192383 time for calcul the mask position with numpy : 0.03193378448486328 nb_pixel_total : 2654 time to create 1 rle with old method : 0.003099679946899414 time for calcul the mask position with numpy : 0.032868385314941406 nb_pixel_total : 59816 time to create 1 rle with old method : 0.06610798835754395 time for calcul the mask position with numpy : 0.02998518943786621 nb_pixel_total : 12482 time to create 1 rle with old method : 0.014075040817260742 time for calcul the mask position with numpy : 0.030678987503051758 nb_pixel_total : 7370 time to create 1 rle with old method : 0.008363723754882812 time for calcul the mask position with numpy : 0.0313260555267334 nb_pixel_total : 25201 time to create 1 rle with old method : 0.028825998306274414 time for calcul the mask position with numpy : 0.0317230224609375 nb_pixel_total : 2652 time to create 1 rle with old method : 0.003082275390625 create new chi : 3.5371296405792236 time to delete rle : 0.001512289047241211 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 10496 TO DO : save crop sub photo not yet done ! save time : 0.6164877414703369 nb_obj : 12 nb_hashtags : 7 time to prepare the origin masks : 10.163408756256104 time for calcul the mask position with numpy : 0.4206719398498535 nb_pixel_total : 6384217 time to create 1 rle with new method : 0.5572261810302734 time for calcul the mask position with numpy : 0.04587197303771973 nb_pixel_total : 10847 time to create 1 rle with old method : 0.011974573135375977 time for calcul the mask position with numpy : 0.04205679893493652 nb_pixel_total : 18397 time to create 1 rle with old method : 0.021230697631835938 time for calcul the mask position with numpy : 0.04458022117614746 nb_pixel_total : 125804 time to create 1 rle with old method : 0.14511704444885254 time for calcul the mask position with numpy : 0.04227733612060547 nb_pixel_total : 2411 time to create 1 rle with old method : 0.0031316280364990234 time for calcul the mask position with numpy : 0.037107229232788086 nb_pixel_total : 81924 time to create 1 rle with old method : 0.08876657485961914 time for calcul the mask position with numpy : 0.026297330856323242 nb_pixel_total : 50443 time to create 1 rle with old method : 0.05674862861633301 time for calcul the mask position with numpy : 0.026537656784057617 nb_pixel_total : 3454 time to create 1 rle with old method : 0.00391387939453125 time for calcul the mask position with numpy : 0.027916431427001953 nb_pixel_total : 956 time to create 1 rle with old method : 0.0010733604431152344 time for calcul the mask position with numpy : 0.02384042739868164 nb_pixel_total : 3052 time to create 1 rle with old method : 0.003509998321533203 time for calcul the mask position with numpy : 0.026857614517211914 nb_pixel_total : 62258 time to create 1 rle with old method : 0.06788039207458496 time for calcul the mask position with numpy : 0.027037858963012695 nb_pixel_total : 9042 time to create 1 rle with old method : 0.009831666946411133 time for calcul the mask position with numpy : 0.02929067611694336 nb_pixel_total : 297435 time to create 1 rle with new method : 0.4181222915649414 create new chi : 2.28068208694458 time to delete rle : 0.001027822494506836 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 8398 TO DO : save crop sub photo not yet done ! save time : 0.5364863872528076 nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 5.527848482131958 time for calcul the mask position with numpy : 0.1959700584411621 nb_pixel_total : 3309502 time to create 1 rle with new method : 1.165755033493042 time for calcul the mask position with numpy : 0.02422475814819336 nb_pixel_total : 28581 time to create 1 rle with old method : 0.03593182563781738 time for calcul the mask position with numpy : 0.026076555252075195 nb_pixel_total : 17353 time to create 1 rle with old method : 0.021979808807373047 time for calcul the mask position with numpy : 0.027124404907226562 nb_pixel_total : 42754 time to create 1 rle with old method : 0.049490928649902344 time for calcul the mask position with numpy : 0.2772023677825928 nb_pixel_total : 3523659 time to create 1 rle with new method : 0.5556557178497314 time for calcul the mask position with numpy : 0.022332191467285156 nb_pixel_total : 41497 time to create 1 rle with old method : 0.049346923828125 time for calcul the mask position with numpy : 0.02461552619934082 nb_pixel_total : 15034 time to create 1 rle with old method : 0.017773151397705078 time for calcul the mask position with numpy : 0.024014711380004883 nb_pixel_total : 71860 time to create 1 rle with old method : 0.08044242858886719 create new chi : 2.6675853729248047 time to delete rle : 0.0010213851928710938 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 10220 TO DO : save crop sub photo not yet done ! save time : 0.6347053050994873 map_output_result : {1339695618: (0.0, 'Should be the crop_list due to order', 0), 1339695553: (0.0, 'Should be the crop_list due to order', 0), 1339695551: (0.0, 'Should be the crop_list due to order', 0), 1339695550: (0.0, 'Should be the crop_list due to order', 0), 1339695548: (0.0, 'Should be the crop_list due to order', 0), 1339695545: (0.0, 'Should be the crop_list due to order', 0), 1339695498: (0.0, 'Should be the crop_list due to order', 0), 1339695497: (0.0, 'Should be the crop_list due to order', 0), 1339695496: (0.0, 'Should be the crop_list due to order', 0), 1339695495: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1339695618, 1339695553, 1339695551, 1339695550, 1339695548, 1339695545, 1339695498, 1339695497, 1339695496, 1339695495] Looping around the photos to save general results len do output : 10 /1339695618.Didn't retrieve data . /1339695553.Didn't retrieve data . /1339695551.Didn't retrieve data . /1339695550.Didn't retrieve data . /1339695548.Didn't retrieve data . /1339695545.Didn't retrieve data . /1339695498.Didn't retrieve data . /1339695497.Didn't retrieve data . /1339695496.Didn't retrieve data . /1339695495.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695618', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695553', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695551', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695550', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695548', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695545', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695498', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695497', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695496', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695495', None, None, None, None, None, '2615673') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014546871185302734 save_final save missing photos in datou_result : time spend for datou_step_exec : 136.46099734306335 time spend to save output : 0.04269075393676758 total time spend for step 5 : 136.50368809700012 step6:crop_condition Wed Feb 26 11:42:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 125 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740566577_1906744 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.481369972229004 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740566582_1906744 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.1597473621368408 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740566584_1906744 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.1889803409576416 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740566588_1906744 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.6154296398162842 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740566592_1906744 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9482040405273438 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740566594_1906744 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5368180274963379 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740566597_1906744 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.0108025074005127 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1339695618, 1339695553, 1339695551, 1339695550, 1339695548, 1339695545, 1339695498, 1339695497, 1339695496, 1339695495] Looping around the photos to save general results len do output : 24 /1339711710Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711711Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711712Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711713Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711719Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711720Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711721Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711722Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711727Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711728Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711729Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711730Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711737Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711738Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711740Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711741Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711742Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711743Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711747Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711748Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711749Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711753Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711756Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339711757Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695618', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695553', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695551', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695550', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695548', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695545', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695498', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695497', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695496', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695495', None, None, None, None, None, '2615673') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 82 time used for this insertion : 0.02856302261352539 save_final save missing photos in datou_result : time spend for datou_step_exec : 24.380091905593872 time spend to save output : 0.036896705627441406 total time spend for step 6 : 24.416988611221313 step7:ventilate_hashtags_in_portfolio Wed Feb 26 11:43:18 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20835307 get user id for portfolio 20835307 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20835307 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('autre_refus','mal_croppe','metal','cartonnette','Teint_Dans_La_Masse','Carton_gris','kraft','plastique','flou','environnement','papier','Carton_brun')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20835307 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('autre_refus','mal_croppe','metal','cartonnette','Teint_Dans_La_Masse','Carton_gris','kraft','plastique','flou','environnement','papier','Carton_brun')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20835752,20835753,20835754,20835755,20835756,20835757,20835758,20835759,20835760,20835761,20835762,20835763?tags=papier,Carton_gris,Carton_brun,Teint_Dans_La_Masse,kraft,cartonnette,metal,autre_refus,mal_croppe,flou,plastique,environnement Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1339695618, 1339695553, 1339695551, 1339695550, 1339695548, 1339695545, 1339695498, 1339695497, 1339695496, 1339695495] Looping around the photos to save general results len do output : 1 /20835307. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695618', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695553', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695551', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695550', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695548', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695545', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695498', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695497', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695496', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695495', None, None, None, None, None, '2615673') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013546228408813477 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.4406156539916992 time spend to save output : 0.01389002799987793 total time spend for step 7 : 1.4545056819915771 step8:final Wed Feb 26 11:43:19 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1339695618: ('0.06609882976282877',), 1339695553: ('0.06609882976282877',), 1339695551: ('0.06609882976282877',), 1339695550: ('0.06609882976282877',), 1339695548: ('0.06609882976282877',), 1339695545: ('0.06609882976282877',), 1339695498: ('0.06609882976282877',), 1339695497: ('0.06609882976282877',), 1339695496: ('0.06609882976282877',), 1339695495: ('0.06609882976282877',)} new output for save of step final : {1339695618: ('0.06609882976282877',), 1339695553: ('0.06609882976282877',), 1339695551: ('0.06609882976282877',), 1339695550: ('0.06609882976282877',), 1339695548: ('0.06609882976282877',), 1339695545: ('0.06609882976282877',), 1339695498: ('0.06609882976282877',), 1339695497: ('0.06609882976282877',), 1339695496: ('0.06609882976282877',), 1339695495: ('0.06609882976282877',)} [1339695618, 1339695553, 1339695551, 1339695550, 1339695548, 1339695545, 1339695498, 1339695497, 1339695496, 1339695495] Looping around the photos to save general results len do output : 10 /1339695618.Didn't retrieve data . /1339695553.Didn't retrieve data . /1339695551.Didn't retrieve data . /1339695550.Didn't retrieve data . /1339695548.Didn't retrieve data . /1339695545.Didn't retrieve data . /1339695498.Didn't retrieve data . /1339695497.Didn't retrieve data . /1339695496.Didn't retrieve data . /1339695495.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695618', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695553', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695551', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695550', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695548', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695545', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695498', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695497', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695496', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695495', None, None, None, None, None, '2615673') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013514041900634766 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10800361633300781 time spend to save output : 0.014174222946166992 total time spend for step 8 : 0.1221778392791748 step9:velours_tree Wed Feb 26 11:43:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.2543008327484131 time spend to save output : 7.176399230957031e-05 total time spend for step 9 : 0.25437259674072266 step10:send_mail_cod Wed Feb 26 11:43:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20835307_26-02-2025_11_43_20.pdf 20835756 change filename to text .change filename to text .change filename to text .imagette208357561740566600 20835757 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208357571740566600 20835758 imagette208357581740566601 20835759 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208357591740566601 20835760 imagette208357601740566601 20835761 imagette208357611740566601 20835762 change filename to text .change filename to text .imagette208357621740566601 20835753 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208357531740566601 20835754 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208357541740566602 20835755 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208357551740566603 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20835307 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20835752,20835753,20835754,20835755,20835756,20835757,20835758,20835759,20835760,20835761,20835762,20835763?tags=papier,Carton_gris,Carton_brun,Teint_Dans_La_Masse,kraft,cartonnette,metal,autre_refus,mal_croppe,flou,plastique,environnement your option no_mail is active, we will not send the real mail to your client args[1339695618] : ('0.06609882976282877',) no score found for photo 1339695618 We are sending mail with results at report@fotonower.com args[1339695553] : ('0.06609882976282877',) no score found for photo 1339695553 We are sending mail with results at report@fotonower.com args[1339695551] : ('0.06609882976282877',) no score found for photo 1339695551 We are sending mail with results at report@fotonower.com args[1339695550] : ('0.06609882976282877',) no score found for photo 1339695550 We are sending mail with results at report@fotonower.com args[1339695548] : ('0.06609882976282877',) no score found for photo 1339695548 We are sending mail with results at report@fotonower.com args[1339695545] : ('0.06609882976282877',) no score found for photo 1339695545 We are sending mail with results at report@fotonower.com args[1339695498] : ('0.06609882976282877',) no score found for photo 1339695498 We are sending mail with results at report@fotonower.com args[1339695497] : ('0.06609882976282877',) no score found for photo 1339695497 We are sending mail with results at report@fotonower.com args[1339695496] : ('0.06609882976282877',) no score found for photo 1339695496 We are sending mail with results at report@fotonower.com args[1339695495] : ('0.06609882976282877',) no score found for photo 1339695495 We are sending mail with results at report@fotonower.com refus_total : 0.06609882976282877 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20835307 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1339695496,1339695498,1339696322,1339695548,1339695550,1339696321,1339695553,1339696320,1339695626,1339695627,1339695682,1339695497,1339695618,1339696323,1339696317,1339695681,1339695628,1339695484,1339695551,1339695545) Found this number of photos: 20 begin to download photo : 1339695496 begin to download photo : 1339696321 begin to download photo : 1339695682 begin to download photo : 1339695681 download finish for photo 1339695496 begin to download photo : 1339695498 download finish for photo 1339696321 begin to download photo : 1339695553 download finish for photo 1339695682 begin to download photo : 1339695497 download finish for photo 1339695681 begin to download photo : 1339695628 download finish for photo 1339695498 begin to download photo : 1339696322 download finish for photo 1339695553 begin to download photo : 1339696320 download finish for photo 1339695628 begin to download photo : 1339695484 download finish for photo 1339695497 begin to download photo : 1339695618 download finish for photo 1339696322 begin to download photo : 1339695548 download finish for photo 1339695484 begin to download photo : 1339695551 download finish for photo 1339696320 begin to download photo : 1339695626 download finish for photo 1339695618 begin to download photo : 1339696323 download finish for photo 1339695548 begin to download photo : 1339695550 download finish for photo 1339695551 begin to download photo : 1339695545 download finish for photo 1339695626 begin to download photo : 1339695627 download finish for photo 1339696323 begin to download photo : 1339696317 download finish for photo 1339695550 download finish for photo 1339696317 download finish for photo 1339695545 download finish for photo 1339695627 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20835307_26-02-2025_11_43_20.pdf results_Auto_P20835307_26-02-2025_11_43_20.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20835307_26-02-2025_11_43_20.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20835307','results_Auto_P20835307_26-02-2025_11_43_20.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20835307_26-02-2025_11_43_20.pdf','pdf','','0.93','0.06609882976282877') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1339695618, 1339695553, 1339695551, 1339695550, 1339695548, 1339695545, 1339695498, 1339695497, 1339695496, 1339695495] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695618', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695553', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695551', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695550', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695548', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695545', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695498', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695497', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695496', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695495', None, None, None, None, None, '2615673') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.013505220413208008 save_final save missing photos in datou_result : time spend for datou_step_exec : 9.269991874694824 time spend to save output : 0.013721227645874023 total time spend for step 10 : 9.283713102340698 step11:split_time_score Wed Feb 26 11:43:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('09', 21),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 26022025 20835307 Nombre de photos uploadées : 21 / 23040 (0%) 26022025 20835307 Nombre de photos taguées (types de déchets): 0 / 21 (0%) 26022025 20835307 Nombre de photos taguées (volume) : 0 / 21 (0%) elapsed_time : load_data_split_time_score 5.4836273193359375e-06 elapsed_time : order_list_meta_photo_and_scores 2.2172927856445312e-05 ????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0014908313751220703 elapsed_time : insert_dashboard_record_day_entry 0.024309873580932617 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20831952_26-02-2025_10_26_15.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20831952 order by id desc limit 1 Qualite : 0.09154895959196861 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20831953_26-02-2025_10_34_24.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20831953 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20831953 AND mptpi.`type`=4207 To do Qualite : 0.07322457552998593 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20831954_26-02-2025_10_18_31.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20831954 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20831954 AND mptpi.`type`=4207 To do Qualite : 0.06367843282466497 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20835305_26-02-2025_11_12_18.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20835305 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20835305 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20835306 order by id desc limit 1 Qualite : 0.06787220550902864 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20835307_26-02-2025_11_43_20.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20835307 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20835307 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20836475 order by id desc limit 1 Qualite : 0.06911020221615603 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20836476_26-02-2025_11_33_19.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20836476 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20836476 AND mptpi.`type`=4207 To do Qualite : 0.0577298937078858 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20836477_26-02-2025_11_36_14.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20836477 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20836477 AND mptpi.`type`=4207 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'26022025': {'nb_upload': 21, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1339695618, 1339695553, 1339695551, 1339695550, 1339695548, 1339695545, 1339695498, 1339695497, 1339695496, 1339695495] Looping around the photos to save general results len do output : 1 /20835307Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695618', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695553', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695551', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695550', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695548', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695545', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695498', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695497', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695496', None, None, None, None, None, '2615673') ('3459', None, None, None, None, None, None, None, '2615673') ('3459', None, '1339695495', None, None, None, None, None, '2615673') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014420509338378906 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.519653081893921 time spend to save output : 0.014853239059448242 total time spend for step 11 : 2.534506320953369 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 136.23user 148.11system 5:09.52elapsed 91%CPU (0avgtext+0avgdata 3963144maxresident)k 2754392inputs+119736outputs (29148major+12059568minor)pagefaults 0swaps