python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2986096 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 16>10 please execute split_portfolio.py -i 21222058 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2654341'] with mtr_portfolio_ids : ['21222058'] and first list_photo_ids : [] new path : /proc/2986096/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.4908909797668457 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Mon Mar 10 14:48:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10151 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-10 14:48:33.813905: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-10 14:48:33.843298: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-10 14:48:33.845013: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f4878000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-10 14:48:33.845034: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-10 14:48:33.847953: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-10 14:48:34.102260: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2a900e50 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-10 14:48:34.102313: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-10 14:48:34.103771: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-10 14:48:34.104158: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-10 14:48:34.106606: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-10 14:48:34.116587: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-10 14:48:34.117444: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-10 14:48:34.122522: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-10 14:48:34.124690: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-10 14:48:34.136438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-10 14:48:34.138213: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-10 14:48:34.138337: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-10 14:48:34.139302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-10 14:48:34.139325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-10 14:48:34.139337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-10 14:48:34.141379: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9399 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-10 14:48:34.628144: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-10 14:48:34.628271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-10 14:48:34.628303: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-10 14:48:34.628333: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-10 14:48:34.628361: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-10 14:48:34.628389: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-10 14:48:34.628415: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-10 14:48:34.628442: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-10 14:48:34.630177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-10 14:48:34.632156: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-10 14:48:34.632267: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-10 14:48:34.632299: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-10 14:48:34.632329: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-10 14:48:34.632360: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-10 14:48:34.632388: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-10 14:48:34.632414: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-10 14:48:34.632443: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-10 14:48:34.634241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-10 14:48:34.634291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-10 14:48:34.634308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-10 14:48:34.634325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-10 14:48:34.636211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9399 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-10 14:48:47.446100: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-10 14:48:47.693497: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 28 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 21 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.07878 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 147.67784 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 24 Detection mask done ! Trying to reset tf kernel 2986655 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5083 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10372 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.6490728855133057 nb_pixel_total : 3533782 time to create 1 rle with new method : 1.124668836593628 length of segment : 2252 time for calcul the mask position with numpy : 0.0002574920654296875 nb_pixel_total : 9880 time to create 1 rle with old method : 0.012640953063964844 length of segment : 110 time for calcul the mask position with numpy : 0.00027680397033691406 nb_pixel_total : 14216 time to create 1 rle with old method : 0.01703166961669922 length of segment : 144 time for calcul the mask position with numpy : 0.0001704692840576172 nb_pixel_total : 8207 time to create 1 rle with old method : 0.0103607177734375 length of segment : 54 time for calcul the mask position with numpy : 0.0003151893615722656 nb_pixel_total : 13256 time to create 1 rle with old method : 0.015881776809692383 length of segment : 161 time for calcul the mask position with numpy : 0.0005855560302734375 nb_pixel_total : 27514 time to create 1 rle with old method : 0.0334935188293457 length of segment : 238 time for calcul the mask position with numpy : 7.557868957519531e-05 nb_pixel_total : 1117 time to create 1 rle with old method : 0.0016345977783203125 length of segment : 40 time for calcul the mask position with numpy : 0.00016045570373535156 nb_pixel_total : 4442 time to create 1 rle with old method : 0.005525350570678711 length of segment : 93 time for calcul the mask position with numpy : 0.0008668899536132812 nb_pixel_total : 43678 time to create 1 rle with old method : 0.04922223091125488 length of segment : 317 time for calcul the mask position with numpy : 0.0014798641204833984 nb_pixel_total : 56002 time to create 1 rle with old method : 0.06762146949768066 length of segment : 525 time for calcul the mask position with numpy : 0.0003249645233154297 nb_pixel_total : 8216 time to create 1 rle with old method : 0.010315895080566406 length of segment : 327 time for calcul the mask position with numpy : 0.000194549560546875 nb_pixel_total : 5352 time to create 1 rle with old method : 0.0067386627197265625 length of segment : 79 time for calcul the mask position with numpy : 0.0001773834228515625 nb_pixel_total : 7630 time to create 1 rle with old method : 0.009412288665771484 length of segment : 93 time for calcul the mask position with numpy : 0.0009107589721679688 nb_pixel_total : 31139 time to create 1 rle with old method : 0.03754019737243652 length of segment : 448 time for calcul the mask position with numpy : 0.0009756088256835938 nb_pixel_total : 56239 time to create 1 rle with old method : 0.06617474555969238 length of segment : 419 time for calcul the mask position with numpy : 0.0019783973693847656 nb_pixel_total : 101248 time to create 1 rle with old method : 0.11828851699829102 length of segment : 360 time for calcul the mask position with numpy : 0.0007061958312988281 nb_pixel_total : 36462 time to create 1 rle with old method : 0.04470968246459961 length of segment : 167 time for calcul the mask position with numpy : 0.00025272369384765625 nb_pixel_total : 13612 time to create 1 rle with old method : 0.017512798309326172 length of segment : 116 time for calcul the mask position with numpy : 0.0002999305725097656 nb_pixel_total : 12023 time to create 1 rle with old method : 0.015090703964233398 length of segment : 97 time for calcul the mask position with numpy : 0.00018358230590820312 nb_pixel_total : 5279 time to create 1 rle with old method : 0.006343364715576172 length of segment : 162 time for calcul the mask position with numpy : 0.00011539459228515625 nb_pixel_total : 2889 time to create 1 rle with old method : 0.003762960433959961 length of segment : 76 time for calcul the mask position with numpy : 8.392333984375e-05 nb_pixel_total : 2183 time to create 1 rle with old method : 0.0029020309448242188 length of segment : 55 time for calcul the mask position with numpy : 0.0030794143676757812 nb_pixel_total : 229178 time to create 1 rle with new method : 0.00667119026184082 length of segment : 424 time for calcul the mask position with numpy : 0.0018112659454345703 nb_pixel_total : 98991 time to create 1 rle with old method : 0.1117696762084961 length of segment : 357 time for calcul the mask position with numpy : 0.00019359588623046875 nb_pixel_total : 6736 time to create 1 rle with old method : 0.008157014846801758 length of segment : 95 time for calcul the mask position with numpy : 0.0024797916412353516 nb_pixel_total : 128385 time to create 1 rle with old method : 0.15259265899658203 length of segment : 405 time for calcul the mask position with numpy : 0.0009913444519042969 nb_pixel_total : 44747 time to create 1 rle with old method : 0.07024264335632324 length of segment : 319 time for calcul the mask position with numpy : 0.002760648727416992 nb_pixel_total : 155276 time to create 1 rle with new method : 0.0042705535888671875 length of segment : 565 time for calcul the mask position with numpy : 0.0013294219970703125 nb_pixel_total : 43186 time to create 1 rle with old method : 0.05145740509033203 length of segment : 337 time for calcul the mask position with numpy : 0.0002334117889404297 nb_pixel_total : 8719 time to create 1 rle with old method : 0.010754108428955078 length of segment : 142 time for calcul the mask position with numpy : 0.0007593631744384766 nb_pixel_total : 26102 time to create 1 rle with old method : 0.030192136764526367 length of segment : 205 time for calcul the mask position with numpy : 0.00436854362487793 nb_pixel_total : 302840 time to create 1 rle with new method : 0.008049249649047852 length of segment : 609 time for calcul the mask position with numpy : 0.0005321502685546875 nb_pixel_total : 25466 time to create 1 rle with old method : 0.031767845153808594 length of segment : 203 time for calcul the mask position with numpy : 0.0004627704620361328 nb_pixel_total : 7721 time to create 1 rle with old method : 0.010004043579101562 length of segment : 136 time for calcul the mask position with numpy : 0.006417989730834961 nb_pixel_total : 288737 time to create 1 rle with new method : 0.009097099304199219 length of segment : 780 time for calcul the mask position with numpy : 0.0014781951904296875 nb_pixel_total : 51751 time to create 1 rle with old method : 0.05965280532836914 length of segment : 488 time for calcul the mask position with numpy : 0.00019788742065429688 nb_pixel_total : 1892 time to create 1 rle with old method : 0.0023567676544189453 length of segment : 90 time for calcul the mask position with numpy : 0.0012938976287841797 nb_pixel_total : 36248 time to create 1 rle with old method : 0.0415341854095459 length of segment : 296 time for calcul the mask position with numpy : 0.0012845993041992188 nb_pixel_total : 33333 time to create 1 rle with old method : 0.05130958557128906 length of segment : 192 time for calcul the mask position with numpy : 0.0015397071838378906 nb_pixel_total : 56995 time to create 1 rle with old method : 0.06745457649230957 length of segment : 322 time for calcul the mask position with numpy : 0.0004248619079589844 nb_pixel_total : 10716 time to create 1 rle with old method : 0.013133049011230469 length of segment : 95 time for calcul the mask position with numpy : 0.001116037368774414 nb_pixel_total : 26985 time to create 1 rle with old method : 0.03158164024353027 length of segment : 402 time for calcul the mask position with numpy : 0.0007276535034179688 nb_pixel_total : 20171 time to create 1 rle with old method : 0.023678302764892578 length of segment : 181 time for calcul the mask position with numpy : 0.0003161430358886719 nb_pixel_total : 4323 time to create 1 rle with old method : 0.005356550216674805 length of segment : 150 time for calcul the mask position with numpy : 0.0023856163024902344 nb_pixel_total : 105861 time to create 1 rle with old method : 0.14537262916564941 length of segment : 336 time for calcul the mask position with numpy : 0.0005507469177246094 nb_pixel_total : 5610 time to create 1 rle with old method : 0.007771492004394531 length of segment : 142 time for calcul the mask position with numpy : 0.0026242733001708984 nb_pixel_total : 112905 time to create 1 rle with old method : 0.13925552368164062 length of segment : 341 time for calcul the mask position with numpy : 0.0003612041473388672 nb_pixel_total : 5433 time to create 1 rle with old method : 0.006837606430053711 length of segment : 136 time for calcul the mask position with numpy : 0.000370025634765625 nb_pixel_total : 2226 time to create 1 rle with old method : 0.0029397010803222656 length of segment : 107 time for calcul the mask position with numpy : 0.0038013458251953125 nb_pixel_total : 169746 time to create 1 rle with new method : 0.00883793830871582 length of segment : 682 time for calcul the mask position with numpy : 0.00017690658569335938 nb_pixel_total : 1402 time to create 1 rle with old method : 0.0017733573913574219 length of segment : 59 time for calcul the mask position with numpy : 0.0012230873107910156 nb_pixel_total : 44139 time to create 1 rle with old method : 0.05542778968811035 length of segment : 337 time for calcul the mask position with numpy : 0.001997709274291992 nb_pixel_total : 54788 time to create 1 rle with old method : 0.06432032585144043 length of segment : 477 time for calcul the mask position with numpy : 0.0033273696899414062 nb_pixel_total : 95239 time to create 1 rle with old method : 0.1263575553894043 length of segment : 390 time for calcul the mask position with numpy : 0.004312276840209961 nb_pixel_total : 144530 time to create 1 rle with old method : 0.17318415641784668 length of segment : 713 time for calcul the mask position with numpy : 0.00021457672119140625 nb_pixel_total : 3798 time to create 1 rle with old method : 0.004855155944824219 length of segment : 66 time for calcul the mask position with numpy : 0.0002772808074951172 nb_pixel_total : 3701 time to create 1 rle with old method : 0.005258321762084961 length of segment : 105 time for calcul the mask position with numpy : 0.0003154277801513672 nb_pixel_total : 3726 time to create 1 rle with old method : 0.005202054977416992 length of segment : 55 time for calcul the mask position with numpy : 0.0017609596252441406 nb_pixel_total : 35276 time to create 1 rle with old method : 0.042638301849365234 length of segment : 422 time for calcul the mask position with numpy : 0.00019240379333496094 nb_pixel_total : 2394 time to create 1 rle with old method : 0.0031349658966064453 length of segment : 46 time for calcul the mask position with numpy : 0.00014162063598632812 nb_pixel_total : 1814 time to create 1 rle with old method : 0.0023488998413085938 length of segment : 54 time for calcul the mask position with numpy : 0.0013513565063476562 nb_pixel_total : 28916 time to create 1 rle with old method : 0.03701138496398926 length of segment : 209 time for calcul the mask position with numpy : 0.0027360916137695312 nb_pixel_total : 77194 time to create 1 rle with old method : 0.09094810485839844 length of segment : 596 time for calcul the mask position with numpy : 0.003050088882446289 nb_pixel_total : 110411 time to create 1 rle with old method : 0.1301121711730957 length of segment : 349 time for calcul the mask position with numpy : 0.005343198776245117 nb_pixel_total : 189347 time to create 1 rle with new method : 0.008414745330810547 length of segment : 445 time for calcul the mask position with numpy : 0.0005667209625244141 nb_pixel_total : 17075 time to create 1 rle with old method : 0.020015478134155273 length of segment : 149 time for calcul the mask position with numpy : 0.007470369338989258 nb_pixel_total : 301005 time to create 1 rle with new method : 0.024784564971923828 length of segment : 686 time for calcul the mask position with numpy : 0.0014050006866455078 nb_pixel_total : 50397 time to create 1 rle with old method : 0.05860733985900879 length of segment : 278 time for calcul the mask position with numpy : 0.0037207603454589844 nb_pixel_total : 115852 time to create 1 rle with old method : 0.13165783882141113 length of segment : 498 time for calcul the mask position with numpy : 0.0010082721710205078 nb_pixel_total : 39240 time to create 1 rle with old method : 0.04478049278259277 length of segment : 261 time for calcul the mask position with numpy : 0.00052642822265625 nb_pixel_total : 7811 time to create 1 rle with old method : 0.009909391403198242 length of segment : 159 time for calcul the mask position with numpy : 0.0008254051208496094 nb_pixel_total : 48598 time to create 1 rle with old method : 0.061460256576538086 length of segment : 228 time for calcul the mask position with numpy : 0.0006282329559326172 nb_pixel_total : 19232 time to create 1 rle with old method : 0.023270130157470703 length of segment : 147 time for calcul the mask position with numpy : 0.0013797283172607422 nb_pixel_total : 28643 time to create 1 rle with old method : 0.0351870059967041 length of segment : 300 time for calcul the mask position with numpy : 0.0005338191986083984 nb_pixel_total : 10237 time to create 1 rle with old method : 0.012066364288330078 length of segment : 125 time for calcul the mask position with numpy : 0.0024445056915283203 nb_pixel_total : 58090 time to create 1 rle with old method : 0.07718610763549805 length of segment : 397 time for calcul the mask position with numpy : 0.0012311935424804688 nb_pixel_total : 53105 time to create 1 rle with old method : 0.05944347381591797 length of segment : 367 time for calcul the mask position with numpy : 0.0005786418914794922 nb_pixel_total : 19004 time to create 1 rle with old method : 0.02309584617614746 length of segment : 175 time for calcul the mask position with numpy : 0.0011005401611328125 nb_pixel_total : 45981 time to create 1 rle with old method : 0.052031755447387695 length of segment : 276 time for calcul the mask position with numpy : 0.001062631607055664 nb_pixel_total : 29137 time to create 1 rle with old method : 0.034813880920410156 length of segment : 305 time for calcul the mask position with numpy : 0.0009424686431884766 nb_pixel_total : 36090 time to create 1 rle with old method : 0.0435023307800293 length of segment : 309 time for calcul the mask position with numpy : 0.00023937225341796875 nb_pixel_total : 5697 time to create 1 rle with old method : 0.00716853141784668 length of segment : 112 time for calcul the mask position with numpy : 0.00043845176696777344 nb_pixel_total : 11028 time to create 1 rle with old method : 0.013505697250366211 length of segment : 148 time for calcul the mask position with numpy : 0.0030634403228759766 nb_pixel_total : 146071 time to create 1 rle with old method : 0.16228842735290527 length of segment : 467 time for calcul the mask position with numpy : 0.0005753040313720703 nb_pixel_total : 12640 time to create 1 rle with old method : 0.015285968780517578 length of segment : 195 time for calcul the mask position with numpy : 0.00022459030151367188 nb_pixel_total : 3834 time to create 1 rle with old method : 0.004792928695678711 length of segment : 88 time for calcul the mask position with numpy : 0.0015332698822021484 nb_pixel_total : 70721 time to create 1 rle with old method : 0.10625314712524414 length of segment : 389 time for calcul the mask position with numpy : 0.0006074905395507812 nb_pixel_total : 20904 time to create 1 rle with old method : 0.025269269943237305 length of segment : 120 time for calcul the mask position with numpy : 0.0008213520050048828 nb_pixel_total : 39050 time to create 1 rle with old method : 0.04508185386657715 length of segment : 220 time for calcul the mask position with numpy : 0.006734609603881836 nb_pixel_total : 337040 time to create 1 rle with new method : 0.013829708099365234 length of segment : 472 time for calcul the mask position with numpy : 0.0007030963897705078 nb_pixel_total : 26912 time to create 1 rle with old method : 0.03119373321533203 length of segment : 155 time for calcul the mask position with numpy : 0.00033020973205566406 nb_pixel_total : 5251 time to create 1 rle with old method : 0.006365060806274414 length of segment : 143 time for calcul the mask position with numpy : 0.0007328987121582031 nb_pixel_total : 28791 time to create 1 rle with old method : 0.03319287300109863 length of segment : 192 time for calcul the mask position with numpy : 0.00038814544677734375 nb_pixel_total : 8497 time to create 1 rle with old method : 0.01034688949584961 length of segment : 138 time for calcul the mask position with numpy : 0.0017857551574707031 nb_pixel_total : 62010 time to create 1 rle with old method : 0.07128334045410156 length of segment : 626 time for calcul the mask position with numpy : 0.00012755393981933594 nb_pixel_total : 1263 time to create 1 rle with old method : 0.001725912094116211 length of segment : 53 time for calcul the mask position with numpy : 0.0017311573028564453 nb_pixel_total : 79447 time to create 1 rle with old method : 0.08989882469177246 length of segment : 461 time for calcul the mask position with numpy : 0.00030303001403808594 nb_pixel_total : 11689 time to create 1 rle with old method : 0.014291048049926758 length of segment : 105 time for calcul the mask position with numpy : 0.00024199485778808594 nb_pixel_total : 3391 time to create 1 rle with old method : 0.004328727722167969 length of segment : 107 time for calcul the mask position with numpy : 0.0017957687377929688 nb_pixel_total : 61380 time to create 1 rle with old method : 0.06973099708557129 length of segment : 379 time for calcul the mask position with numpy : 0.0009226799011230469 nb_pixel_total : 17145 time to create 1 rle with old method : 0.019898176193237305 length of segment : 218 time for calcul the mask position with numpy : 0.0008931159973144531 nb_pixel_total : 22987 time to create 1 rle with old method : 0.02649235725402832 length of segment : 256 time for calcul the mask position with numpy : 0.003139495849609375 nb_pixel_total : 111143 time to create 1 rle with old method : 0.12413978576660156 length of segment : 396 time for calcul the mask position with numpy : 0.0056378841400146484 nb_pixel_total : 242696 time to create 1 rle with new method : 0.007271766662597656 length of segment : 627 time for calcul the mask position with numpy : 0.0032663345336914062 nb_pixel_total : 152600 time to create 1 rle with new method : 0.00390625 length of segment : 535 time for calcul the mask position with numpy : 0.0006983280181884766 nb_pixel_total : 20123 time to create 1 rle with old method : 0.02365732192993164 length of segment : 200 time for calcul the mask position with numpy : 0.003698587417602539 nb_pixel_total : 105397 time to create 1 rle with old method : 0.12121224403381348 length of segment : 506 time for calcul the mask position with numpy : 0.0001575946807861328 nb_pixel_total : 5742 time to create 1 rle with old method : 0.0069310665130615234 length of segment : 94 time for calcul the mask position with numpy : 0.00021266937255859375 nb_pixel_total : 4201 time to create 1 rle with old method : 0.005229473114013672 length of segment : 113 time for calcul the mask position with numpy : 0.0009822845458984375 nb_pixel_total : 15065 time to create 1 rle with old method : 0.01765608787536621 length of segment : 206 time for calcul the mask position with numpy : 0.000232696533203125 nb_pixel_total : 3792 time to create 1 rle with old method : 0.004463911056518555 length of segment : 112 time for calcul the mask position with numpy : 0.0002853870391845703 nb_pixel_total : 4202 time to create 1 rle with old method : 0.0052602291107177734 length of segment : 106 time for calcul the mask position with numpy : 0.00044155120849609375 nb_pixel_total : 8393 time to create 1 rle with old method : 0.01026773452758789 length of segment : 166 time for calcul the mask position with numpy : 0.0005867481231689453 nb_pixel_total : 14703 time to create 1 rle with old method : 0.0173337459564209 length of segment : 140 time for calcul the mask position with numpy : 0.0039789676666259766 nb_pixel_total : 132947 time to create 1 rle with old method : 0.1742234230041504 length of segment : 548 time for calcul the mask position with numpy : 0.00028061866760253906 nb_pixel_total : 10197 time to create 1 rle with old method : 0.012769699096679688 length of segment : 102 time for calcul the mask position with numpy : 0.0022728443145751953 nb_pixel_total : 54728 time to create 1 rle with old method : 0.06578183174133301 length of segment : 365 time for calcul the mask position with numpy : 0.00036144256591796875 nb_pixel_total : 1480 time to create 1 rle with old method : 0.0018601417541503906 length of segment : 68 time for calcul the mask position with numpy : 0.0007266998291015625 nb_pixel_total : 34126 time to create 1 rle with old method : 0.04071855545043945 length of segment : 238 time for calcul the mask position with numpy : 0.0036530494689941406 nb_pixel_total : 134020 time to create 1 rle with old method : 0.14889144897460938 length of segment : 476 time for calcul the mask position with numpy : 0.0002086162567138672 nb_pixel_total : 2393 time to create 1 rle with old method : 0.0028820037841796875 length of segment : 60 time for calcul the mask position with numpy : 0.00027871131896972656 nb_pixel_total : 4169 time to create 1 rle with old method : 0.005187034606933594 length of segment : 99 time for calcul the mask position with numpy : 0.0012576580047607422 nb_pixel_total : 27924 time to create 1 rle with old method : 0.0352170467376709 length of segment : 266 time for calcul the mask position with numpy : 0.0003075599670410156 nb_pixel_total : 4336 time to create 1 rle with old method : 0.005175590515136719 length of segment : 70 time for calcul the mask position with numpy : 0.000438690185546875 nb_pixel_total : 5159 time to create 1 rle with old method : 0.0061511993408203125 length of segment : 127 time for calcul the mask position with numpy : 0.0002758502960205078 nb_pixel_total : 7465 time to create 1 rle with old method : 0.00937509536743164 length of segment : 86 time for calcul the mask position with numpy : 0.0003476142883300781 nb_pixel_total : 3998 time to create 1 rle with old method : 0.005042314529418945 length of segment : 77 time for calcul the mask position with numpy : 0.0021390914916992188 nb_pixel_total : 52676 time to create 1 rle with old method : 0.06346607208251953 length of segment : 346 time for calcul the mask position with numpy : 0.0031070709228515625 nb_pixel_total : 85942 time to create 1 rle with old method : 0.10025405883789062 length of segment : 414 time for calcul the mask position with numpy : 0.0006704330444335938 nb_pixel_total : 13530 time to create 1 rle with old method : 0.016809463500976562 length of segment : 175 time for calcul the mask position with numpy : 0.0007402896881103516 nb_pixel_total : 15504 time to create 1 rle with old method : 0.018591880798339844 length of segment : 157 time for calcul the mask position with numpy : 0.004431486129760742 nb_pixel_total : 142046 time to create 1 rle with old method : 0.17084383964538574 length of segment : 589 time for calcul the mask position with numpy : 0.002026796340942383 nb_pixel_total : 82019 time to create 1 rle with old method : 0.10423111915588379 length of segment : 372 time for calcul the mask position with numpy : 0.0030984878540039062 nb_pixel_total : 59604 time to create 1 rle with old method : 0.07361865043640137 length of segment : 386 time for calcul the mask position with numpy : 0.001026153564453125 nb_pixel_total : 11676 time to create 1 rle with old method : 0.015145063400268555 length of segment : 288 time for calcul the mask position with numpy : 0.0009870529174804688 nb_pixel_total : 16816 time to create 1 rle with old method : 0.02160048484802246 length of segment : 163 time for calcul the mask position with numpy : 0.001177072525024414 nb_pixel_total : 18054 time to create 1 rle with old method : 0.021061182022094727 length of segment : 267 time for calcul the mask position with numpy : 0.0012657642364501953 nb_pixel_total : 26561 time to create 1 rle with old method : 0.03128790855407715 length of segment : 342 time for calcul the mask position with numpy : 0.00018072128295898438 nb_pixel_total : 1451 time to create 1 rle with old method : 0.0017344951629638672 length of segment : 77 time for calcul the mask position with numpy : 0.002295970916748047 nb_pixel_total : 75599 time to create 1 rle with old method : 0.08981466293334961 length of segment : 700 time for calcul the mask position with numpy : 0.0017704963684082031 nb_pixel_total : 31664 time to create 1 rle with old method : 0.03831934928894043 length of segment : 292 time spent for convertir_results : 14.961885452270508 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 141 chid ids of type : 3663 Number RLEs to save : 39248 save missing photos in datou_result : time spend for datou_step_exec : 72.25377178192139 time spend to save output : 5.7374091148376465 total time spend for step 1 : 77.99118089675903 step2:crop_condition Mon Mar 10 14:49:48 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 141 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 45 About to insert : list_path_to_insert length 45 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 12 About to insert : list_path_to_insert length 12 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1342527719, 1342527715, 1342527698, 1342527693, 1342527688, 1342527683, 1342527678, 1342527673, 1342527664, 1342527662] Looping around the photos to save general results len do output : 87 /-3702827505Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827545Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827543Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827562Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827612Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827600Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827607Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827602Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827620Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827531Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827535Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827555Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827575Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827598Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827614Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827606Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827628Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827631Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827640Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827634Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827503Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827526Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827516Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827547Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827577Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827565Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827567Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827584Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827615Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827502Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827518Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827517Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827519Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827510Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827524Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827527Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827506Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827536Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827533Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827532Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827528Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827529Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827539Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827554Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827552Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827540Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827550Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827542Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827573Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827576Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827569Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827570Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827563Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827574Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827566Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827559Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827580Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827587Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827590Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827595Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827617Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827601Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827603Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827605Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827619Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827604Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827616Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827632Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827623Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827641Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827637Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827636Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827633Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827625Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827515Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827509Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827514Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827513Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827553Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827571Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827582Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827583Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827588Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827594Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827610Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827627Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3702827629Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527719', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527715', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527698', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527693', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527688', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527683', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527678', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527673', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527664', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527662', None, None, None, None, None, '2654341') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 271 time used for this insertion : 0.07145333290100098 save_final save missing photos in datou_result : time spend for datou_step_exec : 36.817166566848755 time spend to save output : 0.07634544372558594 total time spend for step 2 : 36.89351201057434 step3:thcl Mon Mar 10 14:50:25 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.1169438362121582 time to convert the images to numpy array : 0.0983128547668457 time to import caffe and check if the image exist : 0.11423540115356445 time to convert the images to numpy array : 0.7395989894866943 time to import caffe and check if the image exist : 0.10130429267883301 time to convert the images to numpy array : 0.7786333560943604 time to import caffe and check if the image exist : 0.10059380531311035 time to convert the images to numpy array : 0.783350944519043 time to import caffe and check if the image exist : 0.13157296180725098 time to convert the images to numpy array : 0.753913402557373 time to import caffe and check if the image exist : 0.12997698783874512 time to convert the images to numpy array : 0.760239839553833 time to import caffe and check if the image exist : 0.10893630981445312 time to convert the images to numpy array : 0.7826030254364014 time to import caffe and check if the image exist : 0.10998964309692383 time to convert the images to numpy array : 0.7835543155670166 time to import caffe and check if the image exist : 0.12384247779846191 time to convert the images to numpy array : 0.7850067615509033 time to import caffe and check if the image exist : 0.09302473068237305 time to convert the images to numpy array : 0.8281822204589844 total time to convert the images to numpy array : 1.3144888877868652 list photo_ids error: [] list photo_ids correct : [-3702827583, -3702827588, -3702827594, -3702827610, -3702827627, -3702827629, -3702827633, -3702827625, -3702827515, -3702827509, -3702827514, -3702827513, -3702827553, -3702827571, -3702827582, -3702827584, -3702827615, -3702827502, -3702827518, -3702827517, -3702827519, -3702827510, -3702827524, -3702827527, -3702827531, -3702827535, -3702827555, -3702827575, -3702827598, -3702827614, -3702827606, -3702827628, -3702827631, -3702827605, -3702827619, -3702827604, -3702827616, -3702827632, -3702827623, -3702827641, -3702827637, -3702827636, -3702827566, -3702827559, -3702827580, -3702827587, -3702827590, -3702827595, -3702827617, -3702827601, -3702827603, -3702827505, -3702827545, -3702827543, -3702827562, -3702827612, -3702827600, -3702827607, -3702827602, -3702827620, -3702827540, -3702827550, -3702827542, -3702827573, -3702827576, -3702827569, -3702827570, -3702827563, -3702827574, -3702827506, -3702827536, -3702827533, -3702827532, -3702827528, -3702827529, -3702827539, -3702827554, -3702827552, -3702827640, -3702827634, -3702827503, -3702827526, -3702827516, -3702827547, -3702827577, -3702827565, -3702827567] number of photos to traite : 87 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10372 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10153 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.9144589900970459 time used to do the prediction : 0.2147371768951416 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10151 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10151 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.9179437160491943 time used to do the prediction : 0.15800905227661133 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.0265579223632812e-05 save missing photos in datou_result : time spend for datou_step_exec : 10.652219772338867 time spend to save output : 0.0008401870727539062 total time spend for step 3 : 10.653059959411621 step4:merge_mask_thcl_custom Mon Mar 10 14:50:36 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 141 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1342527719, 1342527715, 1342527698, 1342527693, 1342527688, 1342527683, 1342527678, 1342527673, 1342527664, 1342527662] Looping around the photos to save general results len do output : 10 /1342527719Didn't retrieve data .Didn't retrieve data . /1342527715Didn't retrieve data .Didn't retrieve data . /1342527698Didn't retrieve data .Didn't retrieve data . /1342527693Didn't retrieve data .Didn't retrieve data . /1342527688Didn't retrieve data .Didn't retrieve data . /1342527683Didn't retrieve data .Didn't retrieve data . /1342527678Didn't retrieve data .Didn't retrieve data . /1342527673Didn't retrieve data .Didn't retrieve data . /1342527664Didn't retrieve data .Didn't retrieve data . /1342527662Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527719', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527715', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527698', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527693', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527688', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527683', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527678', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527673', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527664', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527662', None, None, None, None, None, '2654341') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.13280701637268066 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.19707632064819336 time spend to save output : 0.1333780288696289 total time spend for step 4 : 0.33045434951782227 step5:rle_unique_nms_with_priority Mon Mar 10 14:50:36 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 4 nb_hashtags : 3 time to prepare the origin masks : 4.041517496109009 time for calcul the mask position with numpy : 0.8298797607421875 nb_pixel_total : 3483201 time to create 1 rle with new method : 0.6123106479644775 time for calcul the mask position with numpy : 0.023835420608520508 nb_pixel_total : 8059 time to create 1 rle with old method : 0.009044885635375977 time for calcul the mask position with numpy : 0.02485489845275879 nb_pixel_total : 14186 time to create 1 rle with old method : 0.015735387802124023 time for calcul the mask position with numpy : 0.04085135459899902 nb_pixel_total : 8182 time to create 1 rle with old method : 0.00911259651184082 time for calcul the mask position with numpy : 0.24366068840026855 nb_pixel_total : 3536612 time to create 1 rle with new method : 1.1936554908752441 create new chi : 3.0646281242370605 time to delete rle : 0.07749295234680176 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 7266 TO DO : save crop sub photo not yet done ! save time : 0.818173885345459 nb_obj : 23 nb_hashtags : 6 time to prepare the origin masks : 13.061982154846191 time for calcul the mask position with numpy : 0.6784231662750244 nb_pixel_total : 6308707 time to create 1 rle with new method : 0.606006383895874 time for calcul the mask position with numpy : 0.04391908645629883 nb_pixel_total : 7598 time to create 1 rle with old method : 0.008917093276977539 time for calcul the mask position with numpy : 0.045269012451171875 nb_pixel_total : 76547 time to create 1 rle with old method : 0.08803820610046387 time for calcul the mask position with numpy : 0.04307246208190918 nb_pixel_total : 36260 time to create 1 rle with old method : 0.045385122299194336 time for calcul the mask position with numpy : 0.04352068901062012 nb_pixel_total : 27339 time to create 1 rle with old method : 0.03619980812072754 time for calcul the mask position with numpy : 0.04836392402648926 nb_pixel_total : 55938 time to create 1 rle with old method : 0.062378883361816406 time for calcul the mask position with numpy : 0.04423880577087402 nb_pixel_total : 31093 time to create 1 rle with old method : 0.0422520637512207 time for calcul the mask position with numpy : 0.05146336555480957 nb_pixel_total : 13564 time to create 1 rle with old method : 0.01595902442932129 time for calcul the mask position with numpy : 0.04838109016418457 nb_pixel_total : 6703 time to create 1 rle with old method : 0.007775068283081055 time for calcul the mask position with numpy : 0.055314064025878906 nb_pixel_total : 228708 time to create 1 rle with new method : 0.8518285751342773 time for calcul the mask position with numpy : 0.044081926345825195 nb_pixel_total : 2116 time to create 1 rle with old method : 0.0023992061614990234 time for calcul the mask position with numpy : 0.04513263702392578 nb_pixel_total : 2872 time to create 1 rle with old method : 0.003469705581665039 time for calcul the mask position with numpy : 0.04518723487854004 nb_pixel_total : 5260 time to create 1 rle with old method : 0.00623774528503418 time for calcul the mask position with numpy : 0.045636892318725586 nb_pixel_total : 11957 time to create 1 rle with old method : 0.014252185821533203 time for calcul the mask position with numpy : 0.04729008674621582 nb_pixel_total : 5319 time to create 1 rle with old method : 0.0070953369140625 time for calcul the mask position with numpy : 0.05039072036743164 nb_pixel_total : 8190 time to create 1 rle with old method : 0.009907960891723633 time for calcul the mask position with numpy : 0.04794764518737793 nb_pixel_total : 4422 time to create 1 rle with old method : 0.0053446292877197266 time for calcul the mask position with numpy : 0.04552793502807617 nb_pixel_total : 1242 time to create 1 rle with old method : 0.0016031265258789062 time for calcul the mask position with numpy : 0.05158424377441406 nb_pixel_total : 55869 time to create 1 rle with old method : 0.06470441818237305 time for calcul the mask position with numpy : 0.046671390533447266 nb_pixel_total : 13217 time to create 1 rle with old method : 0.01638007164001465 time for calcul the mask position with numpy : 0.045500755310058594 nb_pixel_total : 44656 time to create 1 rle with old method : 0.05405545234680176 time for calcul the mask position with numpy : 0.04454970359802246 nb_pixel_total : 1114 time to create 1 rle with old method : 0.0015189647674560547 time for calcul the mask position with numpy : 0.04483318328857422 nb_pixel_total : 593 time to create 1 rle with old method : 0.0010387897491455078 time for calcul the mask position with numpy : 0.050901174545288086 nb_pixel_total : 100956 time to create 1 rle with old method : 0.11805558204650879 create new chi : 3.9120476245880127 time to delete rle : 0.0020139217376708984 batch 1 Loaded 24 chid ids of type : 3726 Number RLEs to save : 11826 TO DO : save crop sub photo not yet done ! save time : 1.016352891921997 nb_obj : 10 nb_hashtags : 5 time to prepare the origin masks : 9.198248863220215 time for calcul the mask position with numpy : 0.4676213264465332 nb_pixel_total : 6443922 time to create 1 rle with new method : 0.5892643928527832 time for calcul the mask position with numpy : 0.04676198959350586 nb_pixel_total : 1015 time to create 1 rle with old method : 0.001348733901977539 time for calcul the mask position with numpy : 0.03866124153137207 nb_pixel_total : 43111 time to create 1 rle with old method : 0.05982804298400879 time for calcul the mask position with numpy : 0.04244256019592285 nb_pixel_total : 25045 time to create 1 rle with old method : 0.030140399932861328 time for calcul the mask position with numpy : 0.039641618728637695 nb_pixel_total : 1883 time to create 1 rle with old method : 0.0021772384643554688 time for calcul the mask position with numpy : 0.03770899772644043 nb_pixel_total : 7632 time to create 1 rle with old method : 0.008874654769897461 time for calcul the mask position with numpy : 0.026885032653808594 nb_pixel_total : 9648 time to create 1 rle with old method : 0.011550664901733398 time for calcul the mask position with numpy : 0.04476571083068848 nb_pixel_total : 302547 time to create 1 rle with new method : 0.520256757736206 time for calcul the mask position with numpy : 0.037580013275146484 nb_pixel_total : 8709 time to create 1 rle with old method : 0.00981760025024414 time for calcul the mask position with numpy : 0.04261898994445801 nb_pixel_total : 155213 time to create 1 rle with new method : 0.4999654293060303 time for calcul the mask position with numpy : 0.04006147384643555 nb_pixel_total : 51515 time to create 1 rle with old method : 0.06476736068725586 create new chi : 2.756598949432373 time to delete rle : 0.0015447139739990234 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 7739 TO DO : save crop sub photo not yet done ! save time : 0.9105746746063232 nb_obj : 19 nb_hashtags : 6 time to prepare the origin masks : 15.204453229904175 time for calcul the mask position with numpy : 0.5263679027557373 nb_pixel_total : 6308602 time to create 1 rle with new method : 0.8847439289093018 time for calcul the mask position with numpy : 0.05021500587463379 nb_pixel_total : 44041 time to create 1 rle with old method : 0.05354785919189453 time for calcul the mask position with numpy : 0.042925119400024414 nb_pixel_total : 31906 time to create 1 rle with old method : 0.03389549255371094 time for calcul the mask position with numpy : 0.029882192611694336 nb_pixel_total : 3681 time to create 1 rle with old method : 0.004042863845825195 time for calcul the mask position with numpy : 0.03419184684753418 nb_pixel_total : 7407 time to create 1 rle with old method : 0.010732650756835938 time for calcul the mask position with numpy : 0.03334546089172363 nb_pixel_total : 3736 time to create 1 rle with old method : 0.0040857791900634766 time for calcul the mask position with numpy : 0.03803205490112305 nb_pixel_total : 105419 time to create 1 rle with old method : 0.1188969612121582 time for calcul the mask position with numpy : 0.037224769592285156 nb_pixel_total : 1391 time to create 1 rle with old method : 0.0016396045684814453 time for calcul the mask position with numpy : 0.03455042839050293 nb_pixel_total : 26949 time to create 1 rle with old method : 0.030333995819091797 time for calcul the mask position with numpy : 0.033641815185546875 nb_pixel_total : 6605 time to create 1 rle with old method : 0.008292913436889648 time for calcul the mask position with numpy : 0.0317685604095459 nb_pixel_total : 2227 time to create 1 rle with old method : 0.002727508544921875 time for calcul the mask position with numpy : 0.03427314758300781 nb_pixel_total : 5418 time to create 1 rle with old method : 0.0065155029296875 time for calcul the mask position with numpy : 0.03517746925354004 nb_pixel_total : 54400 time to create 1 rle with old method : 0.07095193862915039 time for calcul the mask position with numpy : 0.030975341796875 nb_pixel_total : 5549 time to create 1 rle with old method : 0.0069427490234375 time for calcul the mask position with numpy : 0.03763532638549805 nb_pixel_total : 144628 time to create 1 rle with old method : 0.18662238121032715 time for calcul the mask position with numpy : 0.055356740951538086 nb_pixel_total : 168252 time to create 1 rle with new method : 0.5083768367767334 time for calcul the mask position with numpy : 0.050522565841674805 nb_pixel_total : 4316 time to create 1 rle with old method : 0.0049016475677490234 time for calcul the mask position with numpy : 0.05273151397705078 nb_pixel_total : 94907 time to create 1 rle with old method : 0.1104269027709961 time for calcul the mask position with numpy : 0.045928001403808594 nb_pixel_total : 10688 time to create 1 rle with old method : 0.011793375015258789 time for calcul the mask position with numpy : 0.04702258110046387 nb_pixel_total : 20118 time to create 1 rle with old method : 0.022101402282714844 create new chi : 3.454227924346924 time to delete rle : 0.0016627311706542969 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 11487 TO DO : save crop sub photo not yet done ! save time : 1.7654194831848145 nb_obj : 19 nb_hashtags : 5 time to prepare the origin masks : 19.846840620040894 time for calcul the mask position with numpy : 0.32767224311828613 nb_pixel_total : 5965940 time to create 1 rle with new method : 1.193725824356079 time for calcul the mask position with numpy : 0.04757523536682129 nb_pixel_total : 35084 time to create 1 rle with old method : 0.04186224937438965 time for calcul the mask position with numpy : 0.05285978317260742 nb_pixel_total : 10032 time to create 1 rle with old method : 0.011417388916015625 time for calcul the mask position with numpy : 0.041486263275146484 nb_pixel_total : 39190 time to create 1 rle with old method : 0.04572486877441406 time for calcul the mask position with numpy : 0.04621005058288574 nb_pixel_total : 286103 time to create 1 rle with new method : 0.770305871963501 time for calcul the mask position with numpy : 0.037085771560668945 nb_pixel_total : 10226 time to create 1 rle with old method : 0.011424541473388672 time for calcul the mask position with numpy : 0.03583860397338867 nb_pixel_total : 28531 time to create 1 rle with old method : 0.031907081604003906 time for calcul the mask position with numpy : 0.03574347496032715 nb_pixel_total : 77296 time to create 1 rle with old method : 0.08678984642028809 time for calcul the mask position with numpy : 0.03545570373535156 nb_pixel_total : 50213 time to create 1 rle with old method : 0.05584001541137695 time for calcul the mask position with numpy : 0.035124778747558594 nb_pixel_total : 115353 time to create 1 rle with old method : 0.13374614715576172 time for calcul the mask position with numpy : 0.03456282615661621 nb_pixel_total : 7803 time to create 1 rle with old method : 0.009023904800415039 time for calcul the mask position with numpy : 0.03528118133544922 nb_pixel_total : 110197 time to create 1 rle with old method : 0.12095427513122559 time for calcul the mask position with numpy : 0.03452801704406738 nb_pixel_total : 52807 time to create 1 rle with old method : 0.05846357345581055 time for calcul the mask position with numpy : 0.03336644172668457 nb_pixel_total : 1803 time to create 1 rle with old method : 0.0020394325256347656 time for calcul the mask position with numpy : 0.021803855895996094 nb_pixel_total : 16987 time to create 1 rle with old method : 0.018921852111816406 time for calcul the mask position with numpy : 0.022202014923095703 nb_pixel_total : 19097 time to create 1 rle with old method : 0.021420955657958984 time for calcul the mask position with numpy : 0.022307872772216797 nb_pixel_total : 28593 time to create 1 rle with old method : 0.03185272216796875 time for calcul the mask position with numpy : 0.02223968505859375 nb_pixel_total : 2385 time to create 1 rle with old method : 0.0027146339416503906 time for calcul the mask position with numpy : 0.021535396575927734 nb_pixel_total : 3695 time to create 1 rle with old method : 0.004420042037963867 time for calcul the mask position with numpy : 0.02513575553894043 nb_pixel_total : 188905 time to create 1 rle with new method : 0.8554074764251709 create new chi : 4.580011606216431 time to delete rle : 0.0016829967498779297 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 12715 TO DO : save crop sub photo not yet done ! save time : 2.4053802490234375 nb_obj : 10 nb_hashtags : 4 time to prepare the origin masks : 8.415553331375122 time for calcul the mask position with numpy : 0.4037363529205322 nb_pixel_total : 6671846 time to create 1 rle with new method : 1.779395341873169 time for calcul the mask position with numpy : 0.04426097869873047 nb_pixel_total : 145689 time to create 1 rle with old method : 0.15918612480163574 time for calcul the mask position with numpy : 0.04333066940307617 nb_pixel_total : 5669 time to create 1 rle with old method : 0.0067005157470703125 time for calcul the mask position with numpy : 0.04156231880187988 nb_pixel_total : 3813 time to create 1 rle with old method : 0.00435185432434082 time for calcul the mask position with numpy : 0.04245448112487793 nb_pixel_total : 12604 time to create 1 rle with old method : 0.013995170593261719 time for calcul the mask position with numpy : 0.04165768623352051 nb_pixel_total : 28800 time to create 1 rle with old method : 0.031679630279541016 time for calcul the mask position with numpy : 0.04108619689941406 nb_pixel_total : 10995 time to create 1 rle with old method : 0.012876510620117188 time for calcul the mask position with numpy : 0.0384058952331543 nb_pixel_total : 35741 time to create 1 rle with old method : 0.03989148139953613 time for calcul the mask position with numpy : 0.0322115421295166 nb_pixel_total : 45837 time to create 1 rle with old method : 0.050989389419555664 time for calcul the mask position with numpy : 0.03145098686218262 nb_pixel_total : 70375 time to create 1 rle with old method : 0.08113670349121094 time for calcul the mask position with numpy : 0.03344297409057617 nb_pixel_total : 18871 time to create 1 rle with old method : 0.02267742156982422 create new chi : 3.0403027534484863 time to delete rle : 0.0010287761688232422 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 7120 TO DO : save crop sub photo not yet done ! save time : 1.2082505226135254 nb_obj : 3 nb_hashtags : 3 time to prepare the origin masks : 2.951677083969116 time for calcul the mask position with numpy : 0.7284555435180664 nb_pixel_total : 6653898 time to create 1 rle with new method : 0.4264991283416748 time for calcul the mask position with numpy : 0.043550968170166016 nb_pixel_total : 20780 time to create 1 rle with old method : 0.025526762008666992 time for calcul the mask position with numpy : 0.03997659683227539 nb_pixel_total : 336629 time to create 1 rle with new method : 0.5005366802215576 time for calcul the mask position with numpy : 0.022072553634643555 nb_pixel_total : 38933 time to create 1 rle with old method : 0.04485917091369629 create new chi : 1.9085140228271484 time to delete rle : 0.0004420280456542969 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 3772 TO DO : save crop sub photo not yet done ! save time : 0.5797445774078369 nb_obj : 7 nb_hashtags : 4 time to prepare the origin masks : 11.120052337646484 time for calcul the mask position with numpy : 2.1381075382232666 nb_pixel_total : 6905720 time to create 1 rle with new method : 0.6087844371795654 time for calcul the mask position with numpy : 0.03872537612915039 nb_pixel_total : 11623 time to create 1 rle with old method : 0.018618345260620117 time for calcul the mask position with numpy : 0.02218794822692871 nb_pixel_total : 1257 time to create 1 rle with old method : 0.0014531612396240234 time for calcul the mask position with numpy : 0.021228313446044922 nb_pixel_total : 8474 time to create 1 rle with old method : 0.009406805038452148 time for calcul the mask position with numpy : 0.03862595558166504 nb_pixel_total : 28631 time to create 1 rle with old method : 0.0330510139465332 time for calcul the mask position with numpy : 0.03633260726928711 nb_pixel_total : 62503 time to create 1 rle with old method : 0.06943607330322266 time for calcul the mask position with numpy : 0.0386958122253418 nb_pixel_total : 5239 time to create 1 rle with old method : 0.00696253776550293 time for calcul the mask position with numpy : 0.03715205192565918 nb_pixel_total : 26793 time to create 1 rle with old method : 0.029310226440429688 create new chi : 3.201558828353882 time to delete rle : 0.0006358623504638672 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 4996 TO DO : save crop sub photo not yet done ! save time : 0.47960591316223145 nb_obj : 20 nb_hashtags : 8 time to prepare the origin masks : 15.704100608825684 time for calcul the mask position with numpy : 0.5622308254241943 nb_pixel_total : 6039313 time to create 1 rle with new method : 0.7332949638366699 time for calcul the mask position with numpy : 0.042807579040527344 nb_pixel_total : 9611 time to create 1 rle with old method : 0.01073598861694336 time for calcul the mask position with numpy : 0.04028201103210449 nb_pixel_total : 4182 time to create 1 rle with old method : 0.004675626754760742 time for calcul the mask position with numpy : 0.04120635986328125 nb_pixel_total : 9532 time to create 1 rle with old method : 0.0108184814453125 time for calcul the mask position with numpy : 0.04269886016845703 nb_pixel_total : 19818 time to create 1 rle with old method : 0.02406167984008789 time for calcul the mask position with numpy : 0.0448453426361084 nb_pixel_total : 241757 time to create 1 rle with new method : 0.8501882553100586 time for calcul the mask position with numpy : 0.04179835319519043 nb_pixel_total : 1472 time to create 1 rle with old method : 0.0017011165618896484 time for calcul the mask position with numpy : 0.04299521446228027 nb_pixel_total : 110992 time to create 1 rle with old method : 0.12201213836669922 time for calcul the mask position with numpy : 0.04128670692443848 nb_pixel_total : 54135 time to create 1 rle with old method : 0.06281113624572754 time for calcul the mask position with numpy : 0.04150509834289551 nb_pixel_total : 15014 time to create 1 rle with old method : 0.017540693283081055 time for calcul the mask position with numpy : 0.04042983055114746 nb_pixel_total : 8359 time to create 1 rle with old method : 0.009760856628417969 time for calcul the mask position with numpy : 0.03892111778259277 nb_pixel_total : 3792 time to create 1 rle with old method : 0.00423741340637207 time for calcul the mask position with numpy : 0.037119150161743164 nb_pixel_total : 33637 time to create 1 rle with old method : 0.03564953804016113 time for calcul the mask position with numpy : 0.038308143615722656 nb_pixel_total : 4177 time to create 1 rle with old method : 0.004465818405151367 time for calcul the mask position with numpy : 0.04028725624084473 nb_pixel_total : 132684 time to create 1 rle with old method : 0.14142227172851562 time for calcul the mask position with numpy : 0.03741145133972168 nb_pixel_total : 14626 time to create 1 rle with old method : 0.01530909538269043 time for calcul the mask position with numpy : 0.03470730781555176 nb_pixel_total : 104688 time to create 1 rle with old method : 0.1078636646270752 time for calcul the mask position with numpy : 0.026092052459716797 nb_pixel_total : 61248 time to create 1 rle with old method : 0.06309270858764648 time for calcul the mask position with numpy : 0.025286197662353516 nb_pixel_total : 5709 time to create 1 rle with old method : 0.006198406219482422 time for calcul the mask position with numpy : 0.023657560348510742 nb_pixel_total : 22969 time to create 1 rle with old method : 0.02451014518737793 time for calcul the mask position with numpy : 0.024170637130737305 nb_pixel_total : 152525 time to create 1 rle with new method : 0.5080969333648682 create new chi : 4.165384531021118 time to delete rle : 0.0016655921936035156 batch 1 Loaded 21 chid ids of type : 3726 Number RLEs to save : 12581 TO DO : save crop sub photo not yet done ! save time : 1.2514715194702148 nb_obj : 22 nb_hashtags : 6 time to prepare the origin masks : 14.332526683807373 time for calcul the mask position with numpy : 0.3092951774597168 nb_pixel_total : 6311530 time to create 1 rle with new method : 0.5440585613250732 time for calcul the mask position with numpy : 0.05131936073303223 nb_pixel_total : 27812 time to create 1 rle with old method : 0.03200125694274902 time for calcul the mask position with numpy : 0.049991607666015625 nb_pixel_total : 16787 time to create 1 rle with old method : 0.017891407012939453 time for calcul the mask position with numpy : 0.04718184471130371 nb_pixel_total : 17959 time to create 1 rle with old method : 0.01910400390625 time for calcul the mask position with numpy : 0.04483318328857422 nb_pixel_total : 5148 time to create 1 rle with old method : 0.0055959224700927734 time for calcul the mask position with numpy : 0.029656171798706055 nb_pixel_total : 1448 time to create 1 rle with old method : 0.0015635490417480469 time for calcul the mask position with numpy : 0.031545162200927734 nb_pixel_total : 4738 time to create 1 rle with old method : 0.0052835941314697266 time for calcul the mask position with numpy : 0.02834463119506836 nb_pixel_total : 11594 time to create 1 rle with old method : 0.012662172317504883 time for calcul the mask position with numpy : 0.04332160949707031 nb_pixel_total : 141943 time to create 1 rle with old method : 0.14981603622436523 time for calcul the mask position with numpy : 0.033403873443603516 nb_pixel_total : 31622 time to create 1 rle with old method : 0.034972190856933594 time for calcul the mask position with numpy : 0.03480410575866699 nb_pixel_total : 13471 time to create 1 rle with old method : 0.017614126205444336 time for calcul the mask position with numpy : 0.038462162017822266 nb_pixel_total : 3973 time to create 1 rle with old method : 0.005197763442993164 time for calcul the mask position with numpy : 0.0372462272644043 nb_pixel_total : 75292 time to create 1 rle with old method : 0.09580850601196289 time for calcul the mask position with numpy : 0.03632354736328125 nb_pixel_total : 15462 time to create 1 rle with old method : 0.06577110290527344 time for calcul the mask position with numpy : 0.03463435173034668 nb_pixel_total : 6188 time to create 1 rle with old method : 0.0072193145751953125 time for calcul the mask position with numpy : 0.030348539352416992 nb_pixel_total : 133760 time to create 1 rle with old method : 0.15270400047302246 time for calcul the mask position with numpy : 0.03803873062133789 nb_pixel_total : 4287 time to create 1 rle with old method : 0.0050241947174072266 time for calcul the mask position with numpy : 0.029485225677490234 nb_pixel_total : 52361 time to create 1 rle with old method : 0.055815935134887695 time for calcul the mask position with numpy : 0.029884815216064453 nb_pixel_total : 4162 time to create 1 rle with old method : 0.004408359527587891 time for calcul the mask position with numpy : 0.030504226684570312 nb_pixel_total : 23027 time to create 1 rle with old method : 0.024806737899780273 time for calcul the mask position with numpy : 0.03247499465942383 nb_pixel_total : 2383 time to create 1 rle with old method : 0.0025649070739746094 time for calcul the mask position with numpy : 0.02982306480407715 nb_pixel_total : 59570 time to create 1 rle with old method : 0.06250500679016113 time for calcul the mask position with numpy : 0.028906822204589844 nb_pixel_total : 85723 time to create 1 rle with old method : 0.09340047836303711 create new chi : 2.5588793754577637 time to delete rle : 0.001821756362915039 batch 1 Loaded 23 chid ids of type : 3726 Number RLEs to save : 13179 TO DO : save crop sub photo not yet done ! save time : 1.6395776271820068 map_output_result : {1342527719: (0.0, 'Should be the crop_list due to order', 0), 1342527715: (0.0, 'Should be the crop_list due to order', 0), 1342527698: (0.0, 'Should be the crop_list due to order', 0), 1342527693: (0.0, 'Should be the crop_list due to order', 0), 1342527688: (0.0, 'Should be the crop_list due to order', 0), 1342527683: (0.0, 'Should be the crop_list due to order', 0), 1342527678: (0.0, 'Should be the crop_list due to order', 0), 1342527673: (0.0, 'Should be the crop_list due to order', 0), 1342527664: (0.0, 'Should be the crop_list due to order', 0), 1342527662: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1342527719, 1342527715, 1342527698, 1342527693, 1342527688, 1342527683, 1342527678, 1342527673, 1342527664, 1342527662] Looping around the photos to save general results len do output : 10 /1342527719.Didn't retrieve data . /1342527715.Didn't retrieve data . /1342527698.Didn't retrieve data . /1342527693.Didn't retrieve data . /1342527688.Didn't retrieve data . /1342527683.Didn't retrieve data . /1342527678.Didn't retrieve data . /1342527673.Didn't retrieve data . /1342527664.Didn't retrieve data . /1342527662.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527719', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527715', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527698', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527693', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527688', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527683', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527678', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527673', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527664', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527662', None, None, None, None, None, '2654341') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.04364824295043945 save_final save missing photos in datou_result : time spend for datou_step_exec : 160.4230456352234 time spend to save output : 0.048677921295166016 total time spend for step 5 : 160.47172355651855 step6:crop_condition Mon Mar 10 14:53:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 147 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741614801_2986096 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 5.584811687469482 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741614808_2986096 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.133643388748169 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741614813_2986096 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 5.551236867904663 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741614822_2986096 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.3479135036468506 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741614826_2986096 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 4.967178821563721 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741614833_2986096 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7546000480651855 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1342527719, 1342527715, 1342527698, 1342527693, 1342527688, 1342527683, 1342527678, 1342527673, 1342527664, 1342527662] Looping around the photos to save general results len do output : 27 /1342541138Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541140Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541145Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541148Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541149Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541153Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541174Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541176Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541190Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541192Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541194Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541196Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541198Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541199Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541204Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541206Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541207Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541209Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541211Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541214Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541249Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541251Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541252Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541254Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541255Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541257Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342541271Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527719', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527715', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527698', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527693', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527688', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527683', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527678', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527673', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527664', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527662', None, None, None, None, None, '2654341') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 91 time used for this insertion : 3.4696919918060303 save_final save missing photos in datou_result : time spend for datou_step_exec : 37.61712884902954 time spend to save output : 3.4707703590393066 total time spend for step 6 : 41.08789920806885 step7:ventilate_hashtags_in_portfolio Mon Mar 10 14:53:58 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 21222058 get user id for portfolio 21222058 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21222058 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','mal_croppe','environnement','metal','flou','Carton_brun','plastique','kraft','Carton_gris','cartonnette','Teint_Dans_La_Masse','autre_refus')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21222058 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','mal_croppe','environnement','metal','flou','Carton_brun','plastique','kraft','Carton_gris','cartonnette','Teint_Dans_La_Masse','autre_refus')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21222058 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','mal_croppe','environnement','metal','flou','Carton_brun','plastique','kraft','Carton_gris','cartonnette','Teint_Dans_La_Masse','autre_refus')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/21222285,21222286,21222287,21222288,21222289,21222290,21222291,21222292,21222293,21222294,21222295,21222296?tags=papier,mal_croppe,environnement,metal,flou,Carton_brun,plastique,kraft,Carton_gris,cartonnette,Teint_Dans_La_Masse,autre_refus Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1342527719, 1342527715, 1342527698, 1342527693, 1342527688, 1342527683, 1342527678, 1342527673, 1342527664, 1342527662] Looping around the photos to save general results len do output : 1 /21222058. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527719', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527715', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527698', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527693', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527688', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527683', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527678', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527673', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527664', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527662', None, None, None, None, None, '2654341') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.033743858337402344 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.527588129043579 time spend to save output : 0.03400087356567383 total time spend for step 7 : 4.561589002609253 step8:final Mon Mar 10 14:54:02 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1342527719: ('0.04524214478601736',), 1342527715: ('0.04524214478601736',), 1342527698: ('0.04524214478601736',), 1342527693: ('0.04524214478601736',), 1342527688: ('0.04524214478601736',), 1342527683: ('0.04524214478601736',), 1342527678: ('0.04524214478601736',), 1342527673: ('0.04524214478601736',), 1342527664: ('0.04524214478601736',), 1342527662: ('0.04524214478601736',)} new output for save of step final : {1342527719: ('0.04524214478601736',), 1342527715: ('0.04524214478601736',), 1342527698: ('0.04524214478601736',), 1342527693: ('0.04524214478601736',), 1342527688: ('0.04524214478601736',), 1342527683: ('0.04524214478601736',), 1342527678: ('0.04524214478601736',), 1342527673: ('0.04524214478601736',), 1342527664: ('0.04524214478601736',), 1342527662: ('0.04524214478601736',)} [1342527719, 1342527715, 1342527698, 1342527693, 1342527688, 1342527683, 1342527678, 1342527673, 1342527664, 1342527662] Looping around the photos to save general results len do output : 10 /1342527719.Didn't retrieve data . /1342527715.Didn't retrieve data . /1342527698.Didn't retrieve data . /1342527693.Didn't retrieve data . /1342527688.Didn't retrieve data . /1342527683.Didn't retrieve data . /1342527678.Didn't retrieve data . /1342527673.Didn't retrieve data . /1342527664.Didn't retrieve data . /1342527662.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527719', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527715', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527698', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527693', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527688', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527683', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527678', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527673', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527664', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527662', None, None, None, None, None, '2654341') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.03933238983154297 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.26151132583618164 time spend to save output : 0.03998970985412598 total time spend for step 8 : 0.3015010356903076 step9:velours_tree Mon Mar 10 14:54:03 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.29785799980163574 time spend to save output : 6.961822509765625e-05 total time spend for step 9 : 0.2979276180267334 step10:send_mail_cod Mon Mar 10 14:54:03 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21222058_10-03-2025_14_54_03.pdf 21222286 imagette212222861741614843 21222288 imagette212222881741614843 21222289 imagette212222891741614843 21222291 change filename to text .imagette212222911741614843 21222292 imagette212222921741614843 21222294 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette212222941741614843 21222296 change filename to text .change filename to text .imagette212222961741614843 21222290 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette212222901741614844 21222293 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette212222931741614845 21222295 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette212222951741614845 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21222058 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21222285,21222286,21222287,21222288,21222289,21222290,21222291,21222292,21222293,21222294,21222295,21222296?tags=papier,mal_croppe,environnement,metal,flou,Carton_brun,plastique,kraft,Carton_gris,cartonnette,Teint_Dans_La_Masse,autre_refus your option no_mail is active, we will not send the real mail to your client args[1342527719] : ('0.04524214478601736',) no score found for photo 1342527719 We are sending mail with results at report@fotonower.com args[1342527715] : ('0.04524214478601736',) no score found for photo 1342527715 We are sending mail with results at report@fotonower.com args[1342527698] : ('0.04524214478601736',) no score found for photo 1342527698 We are sending mail with results at report@fotonower.com args[1342527693] : ('0.04524214478601736',) no score found for photo 1342527693 We are sending mail with results at report@fotonower.com args[1342527688] : ('0.04524214478601736',) no score found for photo 1342527688 We are sending mail with results at report@fotonower.com args[1342527683] : ('0.04524214478601736',) no score found for photo 1342527683 We are sending mail with results at report@fotonower.com args[1342527678] : ('0.04524214478601736',) no score found for photo 1342527678 We are sending mail with results at report@fotonower.com args[1342527673] : ('0.04524214478601736',) no score found for photo 1342527673 We are sending mail with results at report@fotonower.com args[1342527664] : ('0.04524214478601736',) no score found for photo 1342527664 We are sending mail with results at report@fotonower.com args[1342527662] : ('0.04524214478601736',) no score found for photo 1342527662 We are sending mail with results at report@fotonower.com refus_total : 0.04524214478601736 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=21222058 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1342526729,1342526783,1342527647,1342527654,1342527658,1342527662,1342527664,1342527673,1342527688,1342527693,1342527698,1342527715,1342526781,1342527678,1342527683,1342527719) Found this number of photos: 16 begin to download photo : 1342526729 begin to download photo : 1342527658 begin to download photo : 1342527688 begin to download photo : 1342526781 download finish for photo 1342527658 begin to download photo : 1342527662 download finish for photo 1342526781 begin to download photo : 1342527678 download finish for photo 1342527688 begin to download photo : 1342527693 download finish for photo 1342526729 begin to download photo : 1342526783 download finish for photo 1342527693 begin to download photo : 1342527698 download finish for photo 1342527678 begin to download photo : 1342527683 download finish for photo 1342526783 begin to download photo : 1342527647 download finish for photo 1342527662 begin to download photo : 1342527664 download finish for photo 1342527698 begin to download photo : 1342527715 download finish for photo 1342527683 begin to download photo : 1342527719 download finish for photo 1342527664 begin to download photo : 1342527673 download finish for photo 1342527715 download finish for photo 1342527647 begin to download photo : 1342527654 download finish for photo 1342527719 download finish for photo 1342527673 download finish for photo 1342527654 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21222058_10-03-2025_14_54_03.pdf results_Auto_P21222058_10-03-2025_14_54_03.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21222058_10-03-2025_14_54_03.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','21222058','results_Auto_P21222058_10-03-2025_14_54_03.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21222058_10-03-2025_14_54_03.pdf','pdf','','0.58','0.04524214478601736') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1342527719, 1342527715, 1342527698, 1342527693, 1342527688, 1342527683, 1342527678, 1342527673, 1342527664, 1342527662] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527719', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527715', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527698', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527693', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527688', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527683', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527678', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527673', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527664', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527662', None, None, None, None, None, '2654341') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.03967118263244629 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.325824737548828 time spend to save output : 0.039968252182006836 total time spend for step 10 : 6.365792989730835 step11:split_time_score Mon Mar 10 14:54:09 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('12', 9), ('11', 7)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 10032025 21222058 Nombre de photos uploadées : 16 / 23040 (0%) 10032025 21222058 Nombre de photos taguées (types de déchets): 0 / 16 (0%) 10032025 21222058 Nombre de photos taguées (volume) : 0 / 16 (0%) elapsed_time : load_data_split_time_score 7.62939453125e-06 elapsed_time : order_list_meta_photo_and_scores 2.2172927856445312e-05 ???????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0037355422973632812 elapsed_time : insert_dashboard_record_day_entry 0.07428526878356934 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.07314688487932137 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21220680_10-03-2025_14_40_24.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21220680 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21220680 AND mptpi.`type`=4207 To do Qualite : 0.05920215831972459 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21220681_10-03-2025_14_43_23.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21220681 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21220681 AND mptpi.`type`=4207 To do Qualite : 0.08555829745889429 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21220682_10-03-2025_14_15_38.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21220682 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21220682 AND mptpi.`type`=4207 To do Qualite : 0.043012977062255445 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21220683_10-03-2025_08_49_42.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21220683 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21220683 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21222001 order by id desc limit 1 Qualite : 0.08175725035537637 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21222047_10-03-2025_14_52_20.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21222047 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21222047 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21222057 order by id desc limit 1 Qualite : 0.04524214478601736 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21222058_10-03-2025_14_54_03.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21222058 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21222058 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21222084 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21222094 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'10032025': {'nb_upload': 16, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1342527719, 1342527715, 1342527698, 1342527693, 1342527688, 1342527683, 1342527678, 1342527673, 1342527664, 1342527662] Looping around the photos to save general results len do output : 1 /21222058Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527719', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527715', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527698', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527693', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527688', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527683', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527678', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527673', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527664', None, None, None, None, None, '2654341') ('3459', None, None, None, None, None, None, None, '2654341') ('3459', '21222058', '1342527662', None, None, None, None, None, '2654341') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.04343700408935547 save_final save missing photos in datou_result : time spend for datou_step_exec : 5.770366430282593 time spend to save output : 0.04371953010559082 total time spend for step 11 : 5.814085960388184 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 141.16user 156.22system 5:52.97elapsed 84%CPU (0avgtext+0avgdata 4348148maxresident)k 1374944inputs+98288outputs (21089major+13440217minor)pagefaults 0swaps