python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 4234 -l 5 ' -s datou_current_4234 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3572364 load datou : 4234 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) load thcls load THCL from format json or kwargs add thcl : 2847 in CacheModelConfig load THCL from format json or kwargs add thcl : 3237 in CacheModelConfig load pdts add pdt : 5275 in CacheModelConfig add pdt : 5561 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4234, datou_cur_ids : ['2619667'] with mtr_portfolio_ids : ['20881153'] and first list_photo_ids : [] new path : /proc/3572364/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, brightness, blur_detection, rle_unique_nms_with_priority, crop_condition, thcl, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , BFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 2 ; length of list_pids : 2 ; length of list_args : 2 time to download the photos : 0.2279796600341797 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Thu Feb 27 13:20:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10591 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-27 13:20:32.514326: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-27 13:20:32.539270: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-27 13:20:32.541316: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f141c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-27 13:20:32.541349: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-27 13:20:32.545479: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-27 13:20:32.806740: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x41f5e0a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-27 13:20:32.806797: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-27 13:20:32.808082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 13:20:32.808532: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 13:20:32.811696: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 13:20:32.814860: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 13:20:32.815413: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 13:20:32.818570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 13:20:32.819961: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 13:20:32.825367: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 13:20:32.826808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 13:20:32.826884: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 13:20:32.827689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-27 13:20:32.827707: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-27 13:20:32.827716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-27 13:20:32.833649: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9813 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-27 13:20:33.209139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 13:20:33.209250: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 13:20:33.209275: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 13:20:33.209295: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 13:20:33.209317: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 13:20:33.209338: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 13:20:33.209358: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 13:20:33.209378: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 13:20:33.210927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 13:20:33.212484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-27 13:20:33.212538: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-27 13:20:33.212562: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 13:20:33.212583: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-27 13:20:33.212602: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-27 13:20:33.212622: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-27 13:20:33.212640: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-27 13:20:33.212659: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-27 13:20:33.214165: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-27 13:20:33.214207: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-27 13:20:33.214219: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-27 13:20:33.214229: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-27 13:20:33.215817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9813 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2847 thcls : [{'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5275 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) {'thcl': {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'svm_hashtag_type_desc': 5275, 'photo_desc_type': 5275, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_RUBBIA_REFUS_AMIENS_23 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_RUBBIA_REFUS_AMIENS_23 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-27 13:20:41.465752: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-27 13:20:41.707408: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23 /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23/mask_model.h5 size_local : 256009536 size in s3 : 256009536 create time local : 2021-08-09 09:43:22 create time in s3 : 2021-08-06 18:54:04 mask_model.h5 already exist and didn't need to update list_images length : 2 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -120.18438 max: 149.16250 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 0.00012373924255371094 nb_pixel_total : 861 time to create 1 rle with old method : 0.0019643306732177734 length of segment : 48 time for calcul the mask position with numpy : 9.870529174804688e-05 nb_pixel_total : 1041 time to create 1 rle with old method : 0.0023186206817626953 length of segment : 38 time for calcul the mask position with numpy : 0.0003838539123535156 nb_pixel_total : 5193 time to create 1 rle with old method : 0.00716710090637207 length of segment : 122 length of segment : 0 time for calcul the mask position with numpy : 0.000415802001953125 nb_pixel_total : 8011 time to create 1 rle with old method : 0.01856684684753418 length of segment : 218 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.44219 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 6.175041198730469e-05 nb_pixel_total : 1393 time to create 1 rle with old method : 0.0019421577453613281 length of segment : 45 time for calcul the mask position with numpy : 0.00010752677917480469 nb_pixel_total : 2803 time to create 1 rle with old method : 0.0036039352416992188 length of segment : 92 time for calcul the mask position with numpy : 5.14984130859375e-05 nb_pixel_total : 880 time to create 1 rle with old method : 0.0011601448059082031 length of segment : 36 time for calcul the mask position with numpy : 0.00010752677917480469 nb_pixel_total : 3357 time to create 1 rle with old method : 0.0044002532958984375 length of segment : 51 time for calcul the mask position with numpy : 2.8848648071289062e-05 nb_pixel_total : 39 time to create 1 rle with old method : 7.295608520507812e-05 length of segment : 7 time for calcul the mask position with numpy : 8.153915405273438e-05 nb_pixel_total : 3790 time to create 1 rle with old method : 0.0045623779296875 length of segment : 54 time for calcul the mask position with numpy : 8.058547973632812e-05 nb_pixel_total : 2652 time to create 1 rle with old method : 0.0032699108123779297 length of segment : 61 Processing 1 images image shape: (400, 400, 3) min: 1.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -116.94219 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 6.365776062011719e-05 nb_pixel_total : 1510 time to create 1 rle with old method : 0.0021097660064697266 length of segment : 24 time for calcul the mask position with numpy : 9.799003601074219e-05 nb_pixel_total : 5973 time to create 1 rle with old method : 0.007256746292114258 length of segment : 120 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -116.94609 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 0.00015664100646972656 nb_pixel_total : 9842 time to create 1 rle with old method : 0.012279987335205078 length of segment : 88 time for calcul the mask position with numpy : 4.029273986816406e-05 nb_pixel_total : 243 time to create 1 rle with old method : 0.0003464221954345703 length of segment : 22 time for calcul the mask position with numpy : 0.00012636184692382812 nb_pixel_total : 741 time to create 1 rle with old method : 0.0011353492736816406 length of segment : 79 time for calcul the mask position with numpy : 4.553794860839844e-05 nb_pixel_total : 802 time to create 1 rle with old method : 0.000995635986328125 length of segment : 52 time for calcul the mask position with numpy : 4.506111145019531e-05 nb_pixel_total : 1089 time to create 1 rle with old method : 0.0013904571533203125 length of segment : 47 time for calcul the mask position with numpy : 4.482269287109375e-05 nb_pixel_total : 1184 time to create 1 rle with old method : 0.0013954639434814453 length of segment : 44 time for calcul the mask position with numpy : 0.00018405914306640625 nb_pixel_total : 955 time to create 1 rle with old method : 0.002063751220703125 length of segment : 126 time for calcul the mask position with numpy : 5.9604644775390625e-05 nb_pixel_total : 152 time to create 1 rle with old method : 0.00027370452880859375 length of segment : 17 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 6.508827209472656e-05 nb_pixel_total : 536 time to create 1 rle with old method : 0.001068115234375 length of segment : 21 time for calcul the mask position with numpy : 9.226799011230469e-05 nb_pixel_total : 1989 time to create 1 rle with old method : 0.003511190414428711 length of segment : 155 time for calcul the mask position with numpy : 7.62939453125e-05 nb_pixel_total : 1360 time to create 1 rle with old method : 0.0025429725646972656 length of segment : 42 time for calcul the mask position with numpy : 4.935264587402344e-05 nb_pixel_total : 391 time to create 1 rle with old method : 0.0005393028259277344 length of segment : 48 time for calcul the mask position with numpy : 0.0010533332824707031 nb_pixel_total : 62427 time to create 1 rle with old method : 0.07368922233581543 length of segment : 291 time for calcul the mask position with numpy : 0.0006575584411621094 nb_pixel_total : 58181 time to create 1 rle with old method : 0.06328201293945312 length of segment : 253 Processing 1 images image shape: (400, 400, 3) min: 24.00000 max: 178.00000 molded_images shape: (1, 640, 640, 3) min: -92.08281 max: 62.96719 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -119.43438 max: 144.01016 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 5.078315734863281e-05 nb_pixel_total : 230 time to create 1 rle with old method : 0.0004718303680419922 length of segment : 23 time for calcul the mask position with numpy : 4.935264587402344e-05 nb_pixel_total : 671 time to create 1 rle with old method : 0.0011713504791259766 length of segment : 43 time for calcul the mask position with numpy : 0.00011444091796875 nb_pixel_total : 3652 time to create 1 rle with old method : 0.013651132583618164 length of segment : 91 time for calcul the mask position with numpy : 7.176399230957031e-05 nb_pixel_total : 157 time to create 1 rle with old method : 0.0004074573516845703 length of segment : 19 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -115.64922 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 5.602836608886719e-05 nb_pixel_total : 876 time to create 1 rle with old method : 0.0012960433959960938 length of segment : 35 time for calcul the mask position with numpy : 4.458427429199219e-05 nb_pixel_total : 407 time to create 1 rle with old method : 0.0007050037384033203 length of segment : 26 time for calcul the mask position with numpy : 5.1021575927734375e-05 nb_pixel_total : 265 time to create 1 rle with old method : 0.00045871734619140625 length of segment : 19 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -119.50859 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 0.00010395050048828125 nb_pixel_total : 2008 time to create 1 rle with old method : 0.0023033618927001953 length of segment : 191 time for calcul the mask position with numpy : 0.00010037422180175781 nb_pixel_total : 386 time to create 1 rle with old method : 0.000606536865234375 length of segment : 64 time for calcul the mask position with numpy : 4.3392181396484375e-05 nb_pixel_total : 1231 time to create 1 rle with old method : 0.0016248226165771484 length of segment : 40 time for calcul the mask position with numpy : 4.9591064453125e-05 nb_pixel_total : 1206 time to create 1 rle with old method : 0.0016868114471435547 length of segment : 44 time for calcul the mask position with numpy : 4.506111145019531e-05 nb_pixel_total : 419 time to create 1 rle with old method : 0.0006117820739746094 length of segment : 58 time for calcul the mask position with numpy : 0.00011610984802246094 nb_pixel_total : 534 time to create 1 rle with old method : 0.0013885498046875 length of segment : 58 time for calcul the mask position with numpy : 3.743171691894531e-05 nb_pixel_total : 437 time to create 1 rle with old method : 0.0006134510040283203 length of segment : 21 time for calcul the mask position with numpy : 0.0003478527069091797 nb_pixel_total : 7209 time to create 1 rle with old method : 0.008485078811645508 length of segment : 335 time for calcul the mask position with numpy : 4.029273986816406e-05 nb_pixel_total : 850 time to create 1 rle with old method : 0.001004934310913086 length of segment : 38 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 148.25234 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 0.00017118453979492188 nb_pixel_total : 1462 time to create 1 rle with old method : 0.003293752670288086 length of segment : 128 time for calcul the mask position with numpy : 5.650520324707031e-05 nb_pixel_total : 291 time to create 1 rle with old method : 0.0004971027374267578 length of segment : 20 time for calcul the mask position with numpy : 0.00022101402282714844 nb_pixel_total : 3726 time to create 1 rle with old method : 0.00486445426940918 length of segment : 186 time for calcul the mask position with numpy : 0.00011301040649414062 nb_pixel_total : 1384 time to create 1 rle with old method : 0.0019006729125976562 length of segment : 126 time for calcul the mask position with numpy : 4.9114227294921875e-05 nb_pixel_total : 738 time to create 1 rle with old method : 0.0009608268737792969 length of segment : 51 time for calcul the mask position with numpy : 0.00025773048400878906 nb_pixel_total : 5410 time to create 1 rle with old method : 0.007061958312988281 length of segment : 229 time for calcul the mask position with numpy : 5.459785461425781e-05 nb_pixel_total : 788 time to create 1 rle with old method : 0.0010416507720947266 length of segment : 69 time for calcul the mask position with numpy : 0.00020003318786621094 nb_pixel_total : 5430 time to create 1 rle with old method : 0.006711006164550781 length of segment : 228 Processing 1 images image shape: (280, 400, 3) min: 13.00000 max: 185.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 57.21328 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 0.0010941028594970703 nb_pixel_total : 105180 time to create 1 rle with old method : 0.12424325942993164 length of segment : 280 time for calcul the mask position with numpy : 8.487701416015625e-05 nb_pixel_total : 828 time to create 1 rle with old method : 0.001094818115234375 length of segment : 50 time for calcul the mask position with numpy : 5.984306335449219e-05 nb_pixel_total : 901 time to create 1 rle with old method : 0.0012412071228027344 length of segment : 54 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 147.91250 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 10 time for calcul the mask position with numpy : 8.678436279296875e-05 nb_pixel_total : 2844 time to create 1 rle with old method : 0.003688335418701172 length of segment : 47 time for calcul the mask position with numpy : 7.700920104980469e-05 nb_pixel_total : 1494 time to create 1 rle with old method : 0.0022211074829101562 length of segment : 43 time for calcul the mask position with numpy : 6.079673767089844e-05 nb_pixel_total : 1022 time to create 1 rle with old method : 0.001413583755493164 length of segment : 49 time for calcul the mask position with numpy : 5.340576171875e-05 nb_pixel_total : 1076 time to create 1 rle with old method : 0.0015168190002441406 length of segment : 47 time for calcul the mask position with numpy : 0.00010228157043457031 nb_pixel_total : 3736 time to create 1 rle with old method : 0.004851579666137695 length of segment : 77 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 443 time to create 1 rle with old method : 0.0006709098815917969 length of segment : 19 time for calcul the mask position with numpy : 3.5762786865234375e-05 nb_pixel_total : 287 time to create 1 rle with old method : 0.0005125999450683594 length of segment : 10 time for calcul the mask position with numpy : 6.270408630371094e-05 nb_pixel_total : 1159 time to create 1 rle with old method : 0.0016663074493408203 length of segment : 44 time for calcul the mask position with numpy : 0.00044536590576171875 nb_pixel_total : 34486 time to create 1 rle with old method : 0.04122114181518555 length of segment : 247 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 192 time to create 1 rle with old method : 0.00026154518127441406 length of segment : 25 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 244.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 126.86562 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 12 time for calcul the mask position with numpy : 6.246566772460938e-05 nb_pixel_total : 1054 time to create 1 rle with old method : 0.0013911724090576172 length of segment : 38 time for calcul the mask position with numpy : 3.910064697265625e-05 nb_pixel_total : 562 time to create 1 rle with old method : 0.0007894039154052734 length of segment : 31 time for calcul the mask position with numpy : 3.5762786865234375e-05 nb_pixel_total : 379 time to create 1 rle with old method : 0.0005369186401367188 length of segment : 22 time for calcul the mask position with numpy : 4.0531158447265625e-05 nb_pixel_total : 423 time to create 1 rle with old method : 0.0005815029144287109 length of segment : 30 time for calcul the mask position with numpy : 4.220008850097656e-05 nb_pixel_total : 675 time to create 1 rle with old method : 0.0010402202606201172 length of segment : 36 time for calcul the mask position with numpy : 3.6716461181640625e-05 nb_pixel_total : 367 time to create 1 rle with old method : 0.0005171298980712891 length of segment : 30 time for calcul the mask position with numpy : 4.482269287109375e-05 nb_pixel_total : 1040 time to create 1 rle with old method : 0.0013399124145507812 length of segment : 38 time for calcul the mask position with numpy : 4.3392181396484375e-05 nb_pixel_total : 191 time to create 1 rle with old method : 0.0003254413604736328 length of segment : 12 time for calcul the mask position with numpy : 3.933906555175781e-05 nb_pixel_total : 515 time to create 1 rle with old method : 0.0007312297821044922 length of segment : 23 time for calcul the mask position with numpy : 3.695487976074219e-05 nb_pixel_total : 366 time to create 1 rle with old method : 0.0005576610565185547 length of segment : 22 time for calcul the mask position with numpy : 4.8160552978515625e-05 nb_pixel_total : 869 time to create 1 rle with old method : 0.0012416839599609375 length of segment : 32 time for calcul the mask position with numpy : 5.841255187988281e-05 nb_pixel_total : 1076 time to create 1 rle with old method : 0.0014524459838867188 length of segment : 38 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 144.91250 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 5.125999450683594e-05 nb_pixel_total : 341 time to create 1 rle with old method : 0.0006592273712158203 length of segment : 22 time for calcul the mask position with numpy : 4.2438507080078125e-05 nb_pixel_total : 431 time to create 1 rle with old method : 0.0005872249603271484 length of segment : 28 time for calcul the mask position with numpy : 5.221366882324219e-05 nb_pixel_total : 61 time to create 1 rle with old method : 0.00014472007751464844 length of segment : 21 time for calcul the mask position with numpy : 4.2438507080078125e-05 nb_pixel_total : 695 time to create 1 rle with old method : 0.0009913444519042969 length of segment : 27 time for calcul the mask position with numpy : 5.14984130859375e-05 nb_pixel_total : 594 time to create 1 rle with old method : 0.0008349418640136719 length of segment : 43 time for calcul the mask position with numpy : 4.9591064453125e-05 nb_pixel_total : 125 time to create 1 rle with old method : 0.0002422332763671875 length of segment : 24 time for calcul the mask position with numpy : 0.0001671314239501953 nb_pixel_total : 1793 time to create 1 rle with old method : 0.0024461746215820312 length of segment : 130 time for calcul the mask position with numpy : 3.504753112792969e-05 nb_pixel_total : 130 time to create 1 rle with old method : 0.00021791458129882812 length of segment : 11 Processing 1 images image shape: (280, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 137.53750 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 5.14984130859375e-05 nb_pixel_total : 132 time to create 1 rle with old method : 0.00028204917907714844 length of segment : 17 time for calcul the mask position with numpy : 2.8371810913085938e-05 nb_pixel_total : 17 time to create 1 rle with old method : 4.935264587402344e-05 length of segment : 4 time for calcul the mask position with numpy : 3.24249267578125e-05 nb_pixel_total : 261 time to create 1 rle with old method : 0.0003612041473388672 length of segment : 31 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -120.13750 max: 149.05703 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 7.462501525878906e-05 nb_pixel_total : 1073 time to create 1 rle with old method : 0.0022432804107666016 length of segment : 41 time for calcul the mask position with numpy : 5.91278076171875e-05 nb_pixel_total : 709 time to create 1 rle with old method : 0.0015349388122558594 length of segment : 46 time for calcul the mask position with numpy : 7.867813110351562e-05 nb_pixel_total : 2171 time to create 1 rle with old method : 0.004433631896972656 length of segment : 33 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.44219 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 0.00012731552124023438 nb_pixel_total : 2875 time to create 1 rle with old method : 0.0052640438079833984 length of segment : 93 time for calcul the mask position with numpy : 5.6743621826171875e-05 nb_pixel_total : 40 time to create 1 rle with old method : 0.0001227855682373047 length of segment : 7 time for calcul the mask position with numpy : 0.00011467933654785156 nb_pixel_total : 1844 time to create 1 rle with old method : 0.003849029541015625 length of segment : 57 time for calcul the mask position with numpy : 5.507469177246094e-05 nb_pixel_total : 350 time to create 1 rle with old method : 0.0007565021514892578 length of segment : 46 time for calcul the mask position with numpy : 0.00010728836059570312 nb_pixel_total : 2572 time to create 1 rle with old method : 0.005115985870361328 length of segment : 61 time for calcul the mask position with numpy : 0.00012993812561035156 nb_pixel_total : 4023 time to create 1 rle with old method : 0.007980823516845703 length of segment : 71 time for calcul the mask position with numpy : 4.673004150390625e-05 nb_pixel_total : 31 time to create 1 rle with old method : 0.00011658668518066406 length of segment : 5 time for calcul the mask position with numpy : 7.104873657226562e-05 nb_pixel_total : 1592 time to create 1 rle with old method : 0.0032558441162109375 length of segment : 34 time for calcul the mask position with numpy : 0.00010395050048828125 nb_pixel_total : 2612 time to create 1 rle with old method : 0.0044116973876953125 length of segment : 73 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -115.73125 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 8.487701416015625e-05 nb_pixel_total : 1638 time to create 1 rle with old method : 0.0035316944122314453 length of segment : 25 time for calcul the mask position with numpy : 6.413459777832031e-05 nb_pixel_total : 910 time to create 1 rle with old method : 0.0019774436950683594 length of segment : 25 time for calcul the mask position with numpy : 0.00015878677368164062 nb_pixel_total : 5941 time to create 1 rle with old method : 0.011319398880004883 length of segment : 110 time for calcul the mask position with numpy : 0.0003685951232910156 nb_pixel_total : 7340 time to create 1 rle with old method : 0.014551401138305664 length of segment : 233 length of segment : 0 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -116.94219 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 0.000186920166015625 nb_pixel_total : 10019 time to create 1 rle with old method : 0.01766371726989746 length of segment : 89 time for calcul the mask position with numpy : 8.630752563476562e-05 nb_pixel_total : 1243 time to create 1 rle with old method : 0.002710103988647461 length of segment : 44 time for calcul the mask position with numpy : 0.00022602081298828125 nb_pixel_total : 2680 time to create 1 rle with old method : 0.006203651428222656 length of segment : 163 time for calcul the mask position with numpy : 7.104873657226562e-05 nb_pixel_total : 1342 time to create 1 rle with old method : 0.0021142959594726562 length of segment : 46 time for calcul the mask position with numpy : 4.172325134277344e-05 nb_pixel_total : 705 time to create 1 rle with old method : 0.0011780261993408203 length of segment : 52 time for calcul the mask position with numpy : 0.0001456737518310547 nb_pixel_total : 9410 time to create 1 rle with old method : 0.01585221290588379 length of segment : 82 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 4.363059997558594e-05 nb_pixel_total : 520 time to create 1 rle with old method : 0.0007417201995849609 length of segment : 21 time for calcul the mask position with numpy : 7.843971252441406e-05 nb_pixel_total : 2425 time to create 1 rle with old method : 0.0030412673950195312 length of segment : 208 time for calcul the mask position with numpy : 5.316734313964844e-05 nb_pixel_total : 592 time to create 1 rle with old method : 0.0007534027099609375 length of segment : 81 time for calcul the mask position with numpy : 0.00063323974609375 nb_pixel_total : 56068 time to create 1 rle with old method : 0.06405329704284668 length of segment : 244 time for calcul the mask position with numpy : 0.0001304149627685547 nb_pixel_total : 1149 time to create 1 rle with old method : 0.0023195743560791016 length of segment : 39 time for calcul the mask position with numpy : 0.0008578300476074219 nb_pixel_total : 64588 time to create 1 rle with old method : 0.09174418449401855 length of segment : 251 time for calcul the mask position with numpy : 0.00010204315185546875 nb_pixel_total : 633 time to create 1 rle with old method : 0.0008406639099121094 length of segment : 79 Processing 1 images image shape: (400, 400, 3) min: 23.00000 max: 185.00000 molded_images shape: (1, 640, 640, 3) min: -90.16094 max: 62.10391 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 0.001325368881225586 nb_pixel_total : 154124 time to create 1 rle with new method : 0.002015352249145508 length of segment : 409 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -119.45000 max: 138.18203 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 0.00034689903259277344 nb_pixel_total : 18369 time to create 1 rle with old method : 0.02133941650390625 length of segment : 193 time for calcul the mask position with numpy : 3.552436828613281e-05 nb_pixel_total : 188 time to create 1 rle with old method : 0.00027441978454589844 length of segment : 21 time for calcul the mask position with numpy : 8.20159912109375e-05 nb_pixel_total : 4102 time to create 1 rle with old method : 0.005379915237426758 length of segment : 84 time for calcul the mask position with numpy : 3.409385681152344e-05 nb_pixel_total : 216 time to create 1 rle with old method : 0.0002880096435546875 length of segment : 22 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -115.82891 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 4.124641418457031e-05 nb_pixel_total : 500 time to create 1 rle with old method : 0.0007283687591552734 length of segment : 23 time for calcul the mask position with numpy : 3.719329833984375e-05 nb_pixel_total : 304 time to create 1 rle with old method : 0.0005106925964355469 length of segment : 20 time for calcul the mask position with numpy : 3.981590270996094e-05 nb_pixel_total : 563 time to create 1 rle with old method : 0.0008051395416259766 length of segment : 23 time for calcul the mask position with numpy : 3.0279159545898438e-05 nb_pixel_total : 54 time to create 1 rle with old method : 0.00011110305786132812 length of segment : 8 time for calcul the mask position with numpy : 2.9802322387695312e-05 nb_pixel_total : 150 time to create 1 rle with old method : 0.00026154518127441406 length of segment : 16 time for calcul the mask position with numpy : 2.9087066650390625e-05 nb_pixel_total : 60 time to create 1 rle with old method : 0.00011396408081054688 length of segment : 11 time for calcul the mask position with numpy : 6.413459777832031e-05 nb_pixel_total : 2254 time to create 1 rle with old method : 0.0031909942626953125 length of segment : 35 time for calcul the mask position with numpy : 4.38690185546875e-05 nb_pixel_total : 146 time to create 1 rle with old method : 0.0002830028533935547 length of segment : 15 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -119.86797 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 9.989738464355469e-05 nb_pixel_total : 1848 time to create 1 rle with old method : 0.002220630645751953 length of segment : 181 time for calcul the mask position with numpy : 3.8623809814453125e-05 nb_pixel_total : 354 time to create 1 rle with old method : 0.0005464553833007812 length of segment : 18 time for calcul the mask position with numpy : 0.00016021728515625 nb_pixel_total : 47 time to create 1 rle with old method : 0.00013327598571777344 length of segment : 10 time for calcul the mask position with numpy : 5.269050598144531e-05 nb_pixel_total : 1206 time to create 1 rle with old method : 0.002041339874267578 length of segment : 39 time for calcul the mask position with numpy : 3.886222839355469e-05 nb_pixel_total : 794 time to create 1 rle with old method : 0.0012569427490234375 length of segment : 38 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 93 time to create 1 rle with old method : 0.00024175643920898438 length of segment : 7 time for calcul the mask position with numpy : 3.1948089599609375e-05 nb_pixel_total : 299 time to create 1 rle with old method : 0.0005962848663330078 length of segment : 16 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 149.01797 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 7.891654968261719e-05 nb_pixel_total : 1677 time to create 1 rle with old method : 0.002259969711303711 length of segment : 144 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 342 time to create 1 rle with old method : 0.0004858970642089844 length of segment : 23 time for calcul the mask position with numpy : 7.2479248046875e-05 nb_pixel_total : 1491 time to create 1 rle with old method : 0.002088308334350586 length of segment : 134 time for calcul the mask position with numpy : 4.1484832763671875e-05 nb_pixel_total : 662 time to create 1 rle with old method : 0.0008962154388427734 length of segment : 42 time for calcul the mask position with numpy : 0.0002002716064453125 nb_pixel_total : 5400 time to create 1 rle with old method : 0.007013797760009766 length of segment : 233 time for calcul the mask position with numpy : 0.0001556873321533203 nb_pixel_total : 3706 time to create 1 rle with old method : 0.004731893539428711 length of segment : 194 Processing 1 images image shape: (280, 400, 3) min: 10.00000 max: 188.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 63.54531 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 0.0009307861328125 nb_pixel_total : 104672 time to create 1 rle with old method : 0.11939072608947754 length of segment : 283 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 148.98281 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 4.220008850097656e-05 nb_pixel_total : 228 time to create 1 rle with old method : 0.0003540515899658203 length of segment : 17 time for calcul the mask position with numpy : 8.797645568847656e-05 nb_pixel_total : 2260 time to create 1 rle with old method : 0.0035331249237060547 length of segment : 62 time for calcul the mask position with numpy : 3.743171691894531e-05 nb_pixel_total : 267 time to create 1 rle with old method : 0.0004990100860595703 length of segment : 9 time for calcul the mask position with numpy : 3.719329833984375e-05 nb_pixel_total : 454 time to create 1 rle with old method : 0.0005791187286376953 length of segment : 60 time for calcul the mask position with numpy : 5.269050598144531e-05 nb_pixel_total : 1356 time to create 1 rle with old method : 0.0018596649169921875 length of segment : 54 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 247.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 134.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 27 time for calcul the mask position with numpy : 7.081031799316406e-05 nb_pixel_total : 546 time to create 1 rle with old method : 0.0007879734039306641 length of segment : 26 time for calcul the mask position with numpy : 4.5299530029296875e-05 nb_pixel_total : 399 time to create 1 rle with old method : 0.0007877349853515625 length of segment : 17 time for calcul the mask position with numpy : 3.409385681152344e-05 nb_pixel_total : 117 time to create 1 rle with old method : 0.0002086162567138672 length of segment : 13 time for calcul the mask position with numpy : 3.266334533691406e-05 nb_pixel_total : 220 time to create 1 rle with old method : 0.0003104209899902344 length of segment : 19 time for calcul the mask position with numpy : 5.507469177246094e-05 nb_pixel_total : 1029 time to create 1 rle with old method : 0.001425027847290039 length of segment : 44 time for calcul the mask position with numpy : 4.124641418457031e-05 nb_pixel_total : 580 time to create 1 rle with old method : 0.0007336139678955078 length of segment : 29 time for calcul the mask position with numpy : 5.2928924560546875e-05 nb_pixel_total : 1016 time to create 1 rle with old method : 0.0012965202331542969 length of segment : 36 time for calcul the mask position with numpy : 6.508827209472656e-05 nb_pixel_total : 2069 time to create 1 rle with old method : 0.0025746822357177734 length of segment : 63 time for calcul the mask position with numpy : 4.2438507080078125e-05 nb_pixel_total : 641 time to create 1 rle with old method : 0.0009076595306396484 length of segment : 28 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 730 time to create 1 rle with old method : 0.0009334087371826172 length of segment : 35 time for calcul the mask position with numpy : 5.459785461425781e-05 nb_pixel_total : 1298 time to create 1 rle with old method : 0.0017609596252441406 length of segment : 35 time for calcul the mask position with numpy : 6.413459777832031e-05 nb_pixel_total : 2136 time to create 1 rle with old method : 0.002728700637817383 length of segment : 59 time for calcul the mask position with numpy : 3.790855407714844e-05 nb_pixel_total : 255 time to create 1 rle with old method : 0.00037288665771484375 length of segment : 22 time for calcul the mask position with numpy : 4.172325134277344e-05 nb_pixel_total : 669 time to create 1 rle with old method : 0.0008413791656494141 length of segment : 33 time for calcul the mask position with numpy : 3.075599670410156e-05 nb_pixel_total : 129 time to create 1 rle with old method : 0.00019073486328125 length of segment : 13 time for calcul the mask position with numpy : 4.38690185546875e-05 nb_pixel_total : 508 time to create 1 rle with old method : 0.0009837150573730469 length of segment : 36 time for calcul the mask position with numpy : 5.054473876953125e-05 nb_pixel_total : 914 time to create 1 rle with old method : 0.0011813640594482422 length of segment : 32 time for calcul the mask position with numpy : 9.036064147949219e-05 nb_pixel_total : 3022 time to create 1 rle with old method : 0.0047953128814697266 length of segment : 65 time for calcul the mask position with numpy : 6.008148193359375e-05 nb_pixel_total : 748 time to create 1 rle with old method : 0.0009050369262695312 length of segment : 39 time for calcul the mask position with numpy : 5.1021575927734375e-05 nb_pixel_total : 1122 time to create 1 rle with old method : 0.0037441253662109375 length of segment : 33 time for calcul the mask position with numpy : 8.296966552734375e-05 nb_pixel_total : 122 time to create 1 rle with old method : 0.0005125999450683594 length of segment : 14 time for calcul the mask position with numpy : 7.295608520507812e-05 nb_pixel_total : 1055 time to create 1 rle with old method : 0.0031919479370117188 length of segment : 35 time for calcul the mask position with numpy : 0.00016570091247558594 nb_pixel_total : 3291 time to create 1 rle with old method : 0.006108522415161133 length of segment : 73 time for calcul the mask position with numpy : 9.751319885253906e-05 nb_pixel_total : 1208 time to create 1 rle with old method : 0.002370119094848633 length of segment : 36 time for calcul the mask position with numpy : 5.435943603515625e-05 nb_pixel_total : 323 time to create 1 rle with old method : 0.0006923675537109375 length of segment : 21 time for calcul the mask position with numpy : 9.465217590332031e-05 nb_pixel_total : 2602 time to create 1 rle with old method : 0.004258871078491211 length of segment : 87 time for calcul the mask position with numpy : 9.822845458984375e-05 nb_pixel_total : 3040 time to create 1 rle with old method : 0.00496363639831543 length of segment : 93 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 146.60000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 10 time for calcul the mask position with numpy : 4.839897155761719e-05 nb_pixel_total : 177 time to create 1 rle with old method : 0.00025272369384765625 length of segment : 39 time for calcul the mask position with numpy : 0.00011467933654785156 nb_pixel_total : 6928 time to create 1 rle with old method : 0.008702278137207031 length of segment : 70 time for calcul the mask position with numpy : 3.743171691894531e-05 nb_pixel_total : 450 time to create 1 rle with old method : 0.0005612373352050781 length of segment : 27 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 641 time to create 1 rle with old method : 0.0009033679962158203 length of segment : 24 time for calcul the mask position with numpy : 4.1484832763671875e-05 nb_pixel_total : 589 time to create 1 rle with old method : 0.0009431838989257812 length of segment : 43 time for calcul the mask position with numpy : 0.00017261505126953125 nb_pixel_total : 167 time to create 1 rle with old method : 0.00026154518127441406 length of segment : 28 time for calcul the mask position with numpy : 3.24249267578125e-05 nb_pixel_total : 192 time to create 1 rle with old method : 0.00029206275939941406 length of segment : 20 time for calcul the mask position with numpy : 3.337860107421875e-05 nb_pixel_total : 99 time to create 1 rle with old method : 0.00017499923706054688 length of segment : 11 time for calcul the mask position with numpy : 3.3855438232421875e-05 nb_pixel_total : 388 time to create 1 rle with old method : 0.00054931640625 length of segment : 31 time for calcul the mask position with numpy : 5.221366882324219e-05 nb_pixel_total : 653 time to create 1 rle with old method : 0.0009162425994873047 length of segment : 47 Processing 1 images image shape: (280, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 138.22500 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 5.793571472167969e-05 nb_pixel_total : 142 time to create 1 rle with old method : 0.0002777576446533203 length of segment : 17 time for calcul the mask position with numpy : 2.8371810913085938e-05 nb_pixel_total : 20 time to create 1 rle with old method : 5.507469177246094e-05 length of segment : 4 Detection mask done ! Trying to reset tf kernel 3572841 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5083 tf kernel not reseted sub process len(results) : 189 len(list_Values) 189 None max_time_sub_proc : 3600 parent process len(results) : 0 len(list_Values) 189 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10372 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Catched exception ! Connect or reconnect ! Number saved : None batch 1 Loaded 189 chid ids of type : 4228 Number RLEs to save : 12245 save missing photos in datou_result : time spend for datou_step_exec : 25.70802903175354 time spend to save output : 1.2945671081542969 total time spend for step 1 : 27.002596139907837 step2:brightness Thu Feb 27 13:20:56 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step calcul brightness treat image : temp/1740658829_3572364_1339905620_c3785033b8fb122be511208639a1166e.jpg treat image : temp/1740658829_3572364_1339905603_7b255cfb398857e2e71a517b9982c3d0.jpg Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.01466059684753418 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.01150655746459961 save missing photos in datou_result : time spend for datou_step_exec : 0.5424022674560547 time spend to save output : 0.03193521499633789 total time spend for step 2 : 0.5743374824523926 step3:blur_detection Thu Feb 27 13:20:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step blur_detection methode: ratio et variance treat image : temp/1740658829_3572364_1339905620_c3785033b8fb122be511208639a1166e.jpg resize: (1080, 1920) 1339905620 -7.6054192957272955 treat image : temp/1740658829_3572364_1339905603_7b255cfb398857e2e71a517b9982c3d0.jpg resize: (1080, 1920) 1339905603 -7.594240120299947 Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.010938405990600586 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.010849475860595703 save missing photos in datou_result : time spend for datou_step_exec : 1.96311616897583 time spend to save output : 0.026531219482421875 total time spend for step 3 : 1.989647388458252 step4:rle_unique_nms_with_priority Thu Feb 27 13:20:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms batch 1 Loaded 189 chid ids of type : 4228 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++nb_obj : 87 nb_hashtags : 6 time to prepare the origin masks : 1.1568694114685059 time for calcul the mask position with numpy : 0.06136775016784668 nb_pixel_total : 1776946 time to create 1 rle with new method : 0.13338112831115723 time for calcul the mask position with numpy : 0.009018182754516602 nb_pixel_total : 1989 time to create 1 rle with old method : 0.0023412704467773438 time for calcul the mask position with numpy : 0.00952768325805664 nb_pixel_total : 62421 time to create 1 rle with old method : 0.07117724418640137 time for calcul the mask position with numpy : 0.008587360382080078 nb_pixel_total : 1393 time to create 1 rle with old method : 0.0016973018646240234 time for calcul the mask position with numpy : 0.008656978607177734 nb_pixel_total : 1041 time to create 1 rle with old method : 0.0012562274932861328 time for calcul the mask position with numpy : 0.008733510971069336 nb_pixel_total : 861 time to create 1 rle with old method : 0.001004934310913086 time for calcul the mask position with numpy : 0.008487701416015625 nb_pixel_total : 437 time to create 1 rle with old method : 0.0006096363067626953 time for calcul the mask position with numpy : 0.008467912673950195 nb_pixel_total : 3357 time to create 1 rle with old method : 0.003805875778198242 time for calcul the mask position with numpy : 0.008440971374511719 nb_pixel_total : 4250 time to create 1 rle with old method : 0.005101680755615234 time for calcul the mask position with numpy : 0.008535146713256836 nb_pixel_total : 12 time to create 1 rle with old method : 2.8371810913085938e-05 time for calcul the mask position with numpy : 0.008527755737304688 nb_pixel_total : 39 time to create 1 rle with old method : 5.245208740234375e-05 time for calcul the mask position with numpy : 0.008454561233520508 nb_pixel_total : 3674 time to create 1 rle with old method : 0.004269599914550781 time for calcul the mask position with numpy : 0.008553028106689453 nb_pixel_total : 5193 time to create 1 rle with old method : 0.005979776382446289 time for calcul the mask position with numpy : 0.008575439453125 nb_pixel_total : 802 time to create 1 rle with old method : 0.0009810924530029297 time for calcul the mask position with numpy : 0.008752107620239258 nb_pixel_total : 243 time to create 1 rle with old method : 0.0002911090850830078 time for calcul the mask position with numpy : 0.008564472198486328 nb_pixel_total : 2803 time to create 1 rle with old method : 0.0031630992889404297 time for calcul the mask position with numpy : 0.008598566055297852 nb_pixel_total : 391 time to create 1 rle with old method : 0.0004661083221435547 time for calcul the mask position with numpy : 0.008344411849975586 nb_pixel_total : 741 time to create 1 rle with old method : 0.0008721351623535156 time for calcul the mask position with numpy : 0.00977945327758789 nb_pixel_total : 36 time to create 1 rle with old method : 0.00016450881958007812 time for calcul the mask position with numpy : 0.01081085205078125 nb_pixel_total : 661 time to create 1 rle with old method : 0.0011911392211914062 time for calcul the mask position with numpy : 0.010892868041992188 nb_pixel_total : 1510 time to create 1 rle with old method : 0.0027618408203125 time for calcul the mask position with numpy : 0.010874032974243164 nb_pixel_total : 5973 time to create 1 rle with old method : 0.01070261001586914 time for calcul the mask position with numpy : 0.010930538177490234 nb_pixel_total : 152 time to create 1 rle with old method : 0.0003216266632080078 time for calcul the mask position with numpy : 0.011040210723876953 nb_pixel_total : 9842 time to create 1 rle with old method : 0.017540693283081055 time for calcul the mask position with numpy : 0.010941505432128906 nb_pixel_total : 219 time to create 1 rle with old method : 0.0006976127624511719 time for calcul the mask position with numpy : 0.010993480682373047 nb_pixel_total : 1089 time to create 1 rle with old method : 0.0020737648010253906 time for calcul the mask position with numpy : 0.010941028594970703 nb_pixel_total : 1360 time to create 1 rle with old method : 0.0016345977783203125 time for calcul the mask position with numpy : 0.008790731430053711 nb_pixel_total : 536 time to create 1 rle with old method : 0.0006642341613769531 time for calcul the mask position with numpy : 0.008949995040893555 nb_pixel_total : 7209 time to create 1 rle with old method : 0.008896589279174805 time for calcul the mask position with numpy : 0.010524749755859375 nb_pixel_total : 386 time to create 1 rle with old method : 0.0005145072937011719 time for calcul the mask position with numpy : 0.008753299713134766 nb_pixel_total : 850 time to create 1 rle with old method : 0.0009984970092773438 time for calcul the mask position with numpy : 0.008847713470458984 nb_pixel_total : 1231 time to create 1 rle with old method : 0.0014886856079101562 time for calcul the mask position with numpy : 0.008693695068359375 nb_pixel_total : 738 time to create 1 rle with old method : 0.0008943080902099609 time for calcul the mask position with numpy : 0.008515596389770508 nb_pixel_total : 534 time to create 1 rle with old method : 0.0006606578826904297 time for calcul the mask position with numpy : 0.008560419082641602 nb_pixel_total : 265 time to create 1 rle with old method : 0.0003266334533691406 time for calcul the mask position with numpy : 0.008448123931884766 nb_pixel_total : 2008 time to create 1 rle with old method : 0.0023272037506103516 time for calcul the mask position with numpy : 0.008443593978881836 nb_pixel_total : 419 time to create 1 rle with old method : 0.0005135536193847656 time for calcul the mask position with numpy : 0.008428335189819336 nb_pixel_total : 1462 time to create 1 rle with old method : 0.0017132759094238281 time for calcul the mask position with numpy : 0.008950471878051758 nb_pixel_total : 1917 time to create 1 rle with old method : 0.002226114273071289 time for calcul the mask position with numpy : 0.008870363235473633 nb_pixel_total : 291 time to create 1 rle with old method : 0.0005328655242919922 time for calcul the mask position with numpy : 0.010719537734985352 nb_pixel_total : 3726 time to create 1 rle with old method : 0.00457000732421875 time for calcul the mask position with numpy : 0.00846552848815918 nb_pixel_total : 263 time to create 1 rle with old method : 0.00036334991455078125 time for calcul the mask position with numpy : 0.008512496948242188 nb_pixel_total : 170 time to create 1 rle with old method : 0.0002722740173339844 time for calcul the mask position with numpy : 0.008765220642089844 nb_pixel_total : 407 time to create 1 rle with old method : 0.0004849433898925781 time for calcul the mask position with numpy : 0.008410215377807617 nb_pixel_total : 221 time to create 1 rle with old method : 0.0003466606140136719 time for calcul the mask position with numpy : 0.008488178253173828 nb_pixel_total : 437 time to create 1 rle with old method : 0.0005133152008056641 time for calcul the mask position with numpy : 0.008587837219238281 nb_pixel_total : 230 time to create 1 rle with old method : 0.00028324127197265625 time for calcul the mask position with numpy : 0.00854945182800293 nb_pixel_total : 3652 time to create 1 rle with old method : 0.004141092300415039 time for calcul the mask position with numpy : 0.008796215057373047 nb_pixel_total : 1206 time to create 1 rle with old method : 0.0014104843139648438 time for calcul the mask position with numpy : 0.008498668670654297 nb_pixel_total : 157 time to create 1 rle with old method : 0.00023698806762695312 time for calcul the mask position with numpy : 0.008563518524169922 nb_pixel_total : 671 time to create 1 rle with old method : 0.0007536411285400391 time for calcul the mask position with numpy : 0.008433341979980469 nb_pixel_total : 876 time to create 1 rle with old method : 0.0010077953338623047 time for calcul the mask position with numpy : 0.008976221084594727 nb_pixel_total : 105180 time to create 1 rle with old method : 0.12289547920227051 time for calcul the mask position with numpy : 0.008491277694702148 nb_pixel_total : 287 time to create 1 rle with old method : 0.00035643577575683594 time for calcul the mask position with numpy : 0.008908510208129883 nb_pixel_total : 125 time to create 1 rle with old method : 0.0001704692840576172 time for calcul the mask position with numpy : 0.008529186248779297 nb_pixel_total : 26205 time to create 1 rle with old method : 0.030810832977294922 time for calcul the mask position with numpy : 0.00847005844116211 nb_pixel_total : 132 time to create 1 rle with old method : 0.00016832351684570312 time for calcul the mask position with numpy : 0.008455753326416016 nb_pixel_total : 366 time to create 1 rle with old method : 0.0004527568817138672 time for calcul the mask position with numpy : 0.00842142105102539 nb_pixel_total : 191 time to create 1 rle with old method : 0.00023627281188964844 time for calcul the mask position with numpy : 0.008477926254272461 nb_pixel_total : 1076 time to create 1 rle with old method : 0.001233816146850586 time for calcul the mask position with numpy : 0.008488655090332031 nb_pixel_total : 367 time to create 1 rle with old method : 0.0004200935363769531 time for calcul the mask position with numpy : 0.008452892303466797 nb_pixel_total : 2844 time to create 1 rle with old method : 0.0032494068145751953 time for calcul the mask position with numpy : 0.008424043655395508 nb_pixel_total : 379 time to create 1 rle with old method : 0.0004360675811767578 time for calcul the mask position with numpy : 0.008605480194091797 nb_pixel_total : 1793 time to create 1 rle with old method : 0.0027027130126953125 time for calcul the mask position with numpy : 0.008409738540649414 nb_pixel_total : 1076 time to create 1 rle with old method : 0.0012254714965820312 time for calcul the mask position with numpy : 0.008408069610595703 nb_pixel_total : 1054 time to create 1 rle with old method : 0.001194000244140625 time for calcul the mask position with numpy : 0.008421897888183594 nb_pixel_total : 61 time to create 1 rle with old method : 9.036064147949219e-05 time for calcul the mask position with numpy : 0.008471012115478516 nb_pixel_total : 695 time to create 1 rle with old method : 0.0008008480072021484 time for calcul the mask position with numpy : 0.008584260940551758 nb_pixel_total : 3736 time to create 1 rle with old method : 0.0042498111724853516 time for calcul the mask position with numpy : 0.008432626724243164 nb_pixel_total : 1148 time to create 1 rle with old method : 0.0013279914855957031 time for calcul the mask position with numpy : 0.008761405944824219 nb_pixel_total : 341 time to create 1 rle with old method : 0.0004630088806152344 time for calcul the mask position with numpy : 0.009170293807983398 nb_pixel_total : 130 time to create 1 rle with old method : 0.00017547607421875 time for calcul the mask position with numpy : 0.009020328521728516 nb_pixel_total : 1040 time to create 1 rle with old method : 0.001173257827758789 time for calcul the mask position with numpy : 0.008381128311157227 nb_pixel_total : 431 time to create 1 rle with old method : 0.0004870891571044922 time for calcul the mask position with numpy : 0.008366823196411133 nb_pixel_total : 192 time to create 1 rle with old method : 0.00021791458129882812 time for calcul the mask position with numpy : 0.008456230163574219 nb_pixel_total : 443 time to create 1 rle with old method : 0.0005271434783935547 time for calcul the mask position with numpy : 0.008450508117675781 nb_pixel_total : 261 time to create 1 rle with old method : 0.0003085136413574219 time for calcul the mask position with numpy : 0.009513139724731445 nb_pixel_total : 423 time to create 1 rle with old method : 0.0005202293395996094 time for calcul the mask position with numpy : 0.008371591567993164 nb_pixel_total : 17 time to create 1 rle with old method : 3.1948089599609375e-05 time for calcul the mask position with numpy : 0.008380889892578125 nb_pixel_total : 562 time to create 1 rle with old method : 0.0006816387176513672 time for calcul the mask position with numpy : 0.008861064910888672 nb_pixel_total : 515 time to create 1 rle with old method : 0.0006096363067626953 time for calcul the mask position with numpy : 0.008450746536254883 nb_pixel_total : 675 time to create 1 rle with old method : 0.0007867813110351562 time for calcul the mask position with numpy : 0.008463382720947266 nb_pixel_total : 452 time to create 1 rle with old method : 0.0005297660827636719 time for calcul the mask position with numpy : 0.008374452590942383 nb_pixel_total : 594 time to create 1 rle with old method : 0.0006823539733886719 time for calcul the mask position with numpy : 0.008437633514404297 nb_pixel_total : 1022 time to create 1 rle with old method : 0.001657724380493164 time for calcul the mask position with numpy : 0.008421897888183594 nb_pixel_total : 1494 time to create 1 rle with old method : 0.0017633438110351562 time for calcul the mask position with numpy : 0.009258270263671875 nb_pixel_total : 869 time to create 1 rle with old method : 0.0010447502136230469 time for calcul the mask position with numpy : 0.008914709091186523 nb_pixel_total : 127 time to create 1 rle with old method : 0.00018930435180664062 create new chi : 1.342529296875 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.009999275207519531 batch 1 Loaded 88 chid ids of type : 4230 Number RLEs to save : 11395 TO DO : save crop sub photo not yet done ! save time : 1.110745906829834 nb_obj : 100 nb_hashtags : 7 time to prepare the origin masks : 2.1988940238952637 time for calcul the mask position with numpy : 0.25971341133117676 nb_pixel_total : 1629892 time to create 1 rle with new method : 0.23231077194213867 time for calcul the mask position with numpy : 0.007246971130371094 nb_pixel_total : 2572 time to create 1 rle with old method : 0.0029153823852539062 time for calcul the mask position with numpy : 0.006294965744018555 nb_pixel_total : 7340 time to create 1 rle with old method : 0.008268356323242188 time for calcul the mask position with numpy : 0.00639796257019043 nb_pixel_total : 2425 time to create 1 rle with old method : 0.0027358531951904297 time for calcul the mask position with numpy : 0.0064105987548828125 nb_pixel_total : 1073 time to create 1 rle with old method : 0.0012314319610595703 time for calcul the mask position with numpy : 0.00699925422668457 nb_pixel_total : 709 time to create 1 rle with old method : 0.0008504390716552734 time for calcul the mask position with numpy : 0.006896495819091797 nb_pixel_total : 2171 time to create 1 rle with old method : 0.0024614334106445312 time for calcul the mask position with numpy : 0.0066144466400146484 nb_pixel_total : 350 time to create 1 rle with old method : 0.0004165172576904297 time for calcul the mask position with numpy : 0.006406068801879883 nb_pixel_total : 1592 time to create 1 rle with old method : 0.001829385757446289 time for calcul the mask position with numpy : 0.007112264633178711 nb_pixel_total : 31 time to create 1 rle with old method : 5.888938903808594e-05 time for calcul the mask position with numpy : 0.006590127944946289 nb_pixel_total : 19 time to create 1 rle with old method : 9.894371032714844e-05 time for calcul the mask position with numpy : 0.006360530853271484 nb_pixel_total : 8836 time to create 1 rle with old method : 0.01002049446105957 time for calcul the mask position with numpy : 0.007434368133544922 nb_pixel_total : 2198 time to create 1 rle with old method : 0.00267791748046875 time for calcul the mask position with numpy : 0.007073163986206055 nb_pixel_total : 56017 time to create 1 rle with old method : 0.06439089775085449 time for calcul the mask position with numpy : 0.006829023361206055 nb_pixel_total : 40 time to create 1 rle with old method : 6.771087646484375e-05 time for calcul the mask position with numpy : 0.006722450256347656 nb_pixel_total : 1844 time to create 1 rle with old method : 0.002107858657836914 time for calcul the mask position with numpy : 0.006795406341552734 nb_pixel_total : 705 time to create 1 rle with old method : 0.0008034706115722656 time for calcul the mask position with numpy : 0.006421566009521484 nb_pixel_total : 2875 time to create 1 rle with old method : 0.003325223922729492 time for calcul the mask position with numpy : 0.0075817108154296875 nb_pixel_total : 235 time to create 1 rle with old method : 0.0003097057342529297 time for calcul the mask position with numpy : 0.008774518966674805 nb_pixel_total : 16 time to create 1 rle with old method : 4.601478576660156e-05 time for calcul the mask position with numpy : 0.0076808929443359375 nb_pixel_total : 2680 time to create 1 rle with old method : 0.003045797348022461 time for calcul the mask position with numpy : 0.007553815841674805 nb_pixel_total : 910 time to create 1 rle with old method : 0.0012447834014892578 time for calcul the mask position with numpy : 0.008565902709960938 nb_pixel_total : 1638 time to create 1 rle with old method : 0.0018885135650634766 time for calcul the mask position with numpy : 0.00761723518371582 nb_pixel_total : 5941 time to create 1 rle with old method : 0.0065577030181884766 time for calcul the mask position with numpy : 0.008755207061767578 nb_pixel_total : 10019 time to create 1 rle with old method : 0.011173486709594727 time for calcul the mask position with numpy : 0.009039878845214844 nb_pixel_total : 94 time to create 1 rle with old method : 0.00022411346435546875 time for calcul the mask position with numpy : 0.008639812469482422 nb_pixel_total : 105 time to create 1 rle with old method : 0.0001583099365234375 time for calcul the mask position with numpy : 0.007697105407714844 nb_pixel_total : 1243 time to create 1 rle with old method : 0.001382589340209961 time for calcul the mask position with numpy : 0.00834202766418457 nb_pixel_total : 1149 time to create 1 rle with old method : 0.0013020038604736328 time for calcul the mask position with numpy : 0.00870203971862793 nb_pixel_total : 520 time to create 1 rle with old method : 0.0006124973297119141 time for calcul the mask position with numpy : 0.008391618728637695 nb_pixel_total : 154124 time to create 1 rle with new method : 0.05341196060180664 time for calcul the mask position with numpy : 0.010211944580078125 nb_pixel_total : 794 time to create 1 rle with old method : 0.0009448528289794922 time for calcul the mask position with numpy : 0.010147809982299805 nb_pixel_total : 2254 time to create 1 rle with old method : 0.0025932788848876953 time for calcul the mask position with numpy : 0.010046005249023438 nb_pixel_total : 1206 time to create 1 rle with old method : 0.0013380050659179688 time for calcul the mask position with numpy : 0.0102691650390625 nb_pixel_total : 662 time to create 1 rle with old method : 0.0007863044738769531 time for calcul the mask position with numpy : 0.010169267654418945 nb_pixel_total : 18369 time to create 1 rle with old method : 0.02035808563232422 time for calcul the mask position with numpy : 0.010072708129882812 nb_pixel_total : 47 time to create 1 rle with old method : 7.82012939453125e-05 time for calcul the mask position with numpy : 0.009935140609741211 nb_pixel_total : 299 time to create 1 rle with old method : 0.00035452842712402344 time for calcul the mask position with numpy : 0.010004520416259766 nb_pixel_total : 1848 time to create 1 rle with old method : 0.002112865447998047 time for calcul the mask position with numpy : 0.008740425109863281 nb_pixel_total : 1677 time to create 1 rle with old method : 0.0020058155059814453 time for calcul the mask position with numpy : 0.01332712173461914 nb_pixel_total : 5400 time to create 1 rle with old method : 0.008546113967895508 time for calcul the mask position with numpy : 0.012078523635864258 nb_pixel_total : 342 time to create 1 rle with old method : 0.0004305839538574219 time for calcul the mask position with numpy : 0.009523630142211914 nb_pixel_total : 108 time to create 1 rle with old method : 0.00020503997802734375 time for calcul the mask position with numpy : 0.010155439376831055 nb_pixel_total : 318 time to create 1 rle with old method : 0.0004363059997558594 time for calcul the mask position with numpy : 0.009951353073120117 nb_pixel_total : 354 time to create 1 rle with old method : 0.00046133995056152344 time for calcul the mask position with numpy : 0.008887052536010742 nb_pixel_total : 304 time to create 1 rle with old method : 0.0003712177276611328 time for calcul the mask position with numpy : 0.00877070426940918 nb_pixel_total : 28 time to create 1 rle with old method : 5.0067901611328125e-05 time for calcul the mask position with numpy : 0.00883626937866211 nb_pixel_total : 188 time to create 1 rle with old method : 0.00022363662719726562 time for calcul the mask position with numpy : 0.006537199020385742 nb_pixel_total : 205 time to create 1 rle with old method : 0.00031685829162597656 time for calcul the mask position with numpy : 0.006585359573364258 nb_pixel_total : 146 time to create 1 rle with old method : 0.00017833709716796875 time for calcul the mask position with numpy : 0.006541728973388672 nb_pixel_total : 54 time to create 1 rle with old method : 8.559226989746094e-05 time for calcul the mask position with numpy : 0.006513118743896484 nb_pixel_total : 60 time to create 1 rle with old method : 0.00011968612670898438 time for calcul the mask position with numpy : 0.0065288543701171875 nb_pixel_total : 93 time to create 1 rle with old method : 0.00012922286987304688 time for calcul the mask position with numpy : 0.006315946578979492 nb_pixel_total : 67 time to create 1 rle with old method : 0.0001289844512939453 time for calcul the mask position with numpy : 0.006416797637939453 nb_pixel_total : 150 time to create 1 rle with old method : 0.00018835067749023438 time for calcul the mask position with numpy : 0.0064127445220947266 nb_pixel_total : 500 time to create 1 rle with old method : 0.0005879402160644531 time for calcul the mask position with numpy : 0.00728917121887207 nb_pixel_total : 104672 time to create 1 rle with old method : 0.11747336387634277 time for calcul the mask position with numpy : 0.00625157356262207 nb_pixel_total : 267 time to create 1 rle with old method : 0.0003299713134765625 time for calcul the mask position with numpy : 0.006343364715576172 nb_pixel_total : 142 time to create 1 rle with old method : 0.00018024444580078125 time for calcul the mask position with numpy : 0.006547451019287109 nb_pixel_total : 129 time to create 1 rle with old method : 0.0001742839813232422 time for calcul the mask position with numpy : 0.0063970088958740234 nb_pixel_total : 323 time to create 1 rle with old method : 0.00037789344787597656 time for calcul the mask position with numpy : 0.006826877593994141 nb_pixel_total : 748 time to create 1 rle with old method : 0.0009052753448486328 time for calcul the mask position with numpy : 0.006494998931884766 nb_pixel_total : 255 time to create 1 rle with old method : 0.0003609657287597656 time for calcul the mask position with numpy : 0.0065271854400634766 nb_pixel_total : 2260 time to create 1 rle with old method : 0.002648591995239258 time for calcul the mask position with numpy : 0.0067026615142822266 nb_pixel_total : 508 time to create 1 rle with old method : 0.0005807876586914062 time for calcul the mask position with numpy : 0.00646662712097168 nb_pixel_total : 167 time to create 1 rle with old method : 0.000209808349609375 time for calcul the mask position with numpy : 0.006373167037963867 nb_pixel_total : 449 time to create 1 rle with old method : 0.0005943775177001953 time for calcul the mask position with numpy : 0.006434202194213867 nb_pixel_total : 2602 time to create 1 rle with old method : 0.0029573440551757812 time for calcul the mask position with numpy : 0.006350517272949219 nb_pixel_total : 220 time to create 1 rle with old method : 0.0002651214599609375 time for calcul the mask position with numpy : 0.0063664913177490234 nb_pixel_total : 177 time to create 1 rle with old method : 0.00022721290588378906 time for calcul the mask position with numpy : 0.006377696990966797 nb_pixel_total : 914 time to create 1 rle with old method : 0.0010170936584472656 time for calcul the mask position with numpy : 0.006196260452270508 nb_pixel_total : 641 time to create 1 rle with old method : 0.0007426738739013672 time for calcul the mask position with numpy : 0.006208658218383789 nb_pixel_total : 2069 time to create 1 rle with old method : 0.002304553985595703 time for calcul the mask position with numpy : 0.006235837936401367 nb_pixel_total : 454 time to create 1 rle with old method : 0.0005285739898681641 time for calcul the mask position with numpy : 0.006379604339599609 nb_pixel_total : 132 time to create 1 rle with old method : 0.00021886825561523438 time for calcul the mask position with numpy : 0.006310224533081055 nb_pixel_total : 1016 time to create 1 rle with old method : 0.0012211799621582031 time for calcul the mask position with numpy : 0.00628972053527832 nb_pixel_total : 228 time to create 1 rle with old method : 0.00029730796813964844 time for calcul the mask position with numpy : 0.006335258483886719 nb_pixel_total : 99 time to create 1 rle with old method : 0.00013446807861328125 time for calcul the mask position with numpy : 0.0062825679779052734 nb_pixel_total : 119 time to create 1 rle with old method : 0.00015473365783691406 time for calcul the mask position with numpy : 0.00639796257019043 nb_pixel_total : 450 time to create 1 rle with old method : 0.0005195140838623047 time for calcul the mask position with numpy : 0.006335020065307617 nb_pixel_total : 44 time to create 1 rle with old method : 9.965896606445312e-05 time for calcul the mask position with numpy : 0.0063703060150146484 nb_pixel_total : 1122 time to create 1 rle with old method : 0.0012731552124023438 time for calcul the mask position with numpy : 0.006207704544067383 nb_pixel_total : 1 time to create 1 rle with old method : 2.1219253540039062e-05 time for calcul the mask position with numpy : 0.0061321258544921875 nb_pixel_total : 111 time to create 1 rle with old method : 0.00014352798461914062 time for calcul the mask position with numpy : 0.006026029586791992 nb_pixel_total : 399 time to create 1 rle with old method : 0.00045943260192871094 time for calcul the mask position with numpy : 0.006179094314575195 nb_pixel_total : 725 time to create 1 rle with old method : 0.0008578300476074219 time for calcul the mask position with numpy : 0.006110429763793945 nb_pixel_total : 1298 time to create 1 rle with old method : 0.001550912857055664 time for calcul the mask position with numpy : 0.008518218994140625 nb_pixel_total : 20 time to create 1 rle with old method : 3.409385681152344e-05 time for calcul the mask position with numpy : 0.008391380310058594 nb_pixel_total : 117 time to create 1 rle with old method : 0.00014281272888183594 time for calcul the mask position with numpy : 0.008365154266357422 nb_pixel_total : 260 time to create 1 rle with old method : 0.00034427642822265625 time for calcul the mask position with numpy : 0.008377313613891602 nb_pixel_total : 1356 time to create 1 rle with old method : 0.001524209976196289 time for calcul the mask position with numpy : 0.008325338363647461 nb_pixel_total : 3017 time to create 1 rle with old method : 0.0033698081970214844 time for calcul the mask position with numpy : 0.008349418640136719 nb_pixel_total : 580 time to create 1 rle with old method : 0.00064849853515625 time for calcul the mask position with numpy : 0.00838470458984375 nb_pixel_total : 192 time to create 1 rle with old method : 0.00021886825561523438 time for calcul the mask position with numpy : 0.008383512496948242 nb_pixel_total : 669 time to create 1 rle with old method : 0.0007581710815429688 time for calcul the mask position with numpy : 0.011957883834838867 nb_pixel_total : 6928 time to create 1 rle with old method : 0.00778961181640625 time for calcul the mask position with numpy : 0.008393049240112305 nb_pixel_total : 79 time to create 1 rle with old method : 0.0001418590545654297 time for calcul the mask position with numpy : 0.008358955383300781 nb_pixel_total : 589 time to create 1 rle with old method : 0.0006608963012695312 time for calcul the mask position with numpy : 0.008361101150512695 nb_pixel_total : 1029 time to create 1 rle with old method : 0.0011508464813232422 time for calcul the mask position with numpy : 0.008375167846679688 nb_pixel_total : 641 time to create 1 rle with old method : 0.0007736682891845703 time for calcul the mask position with numpy : 0.008789777755737305 nb_pixel_total : 546 time to create 1 rle with old method : 0.0006377696990966797 create new chi : 1.6521146297454834 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.001379251480102539 batch 1 Loaded 101 chid ids of type : 4230 Number RLEs to save : 11747 TO DO : save crop sub photo not yet done ! save time : 0.6972439289093018 map_output_result : {1339905620: (0.0, 'Should be the crop_list due to order', 0.0), 1339905603: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1339905620, 1339905603] Looping around the photos to save general results len do output : 2 /1339905620.Didn't retrieve data . /1339905603.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905620', None, None, None, None, None, '2619667') ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905603', None, None, None, None, None, '2619667') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.013538599014282227 save_final save missing photos in datou_result : time spend for datou_step_exec : 8.566572904586792 time spend to save output : 0.013910531997680664 total time spend for step 4 : 8.580483436584473 step5:crop_condition Thu Feb 27 13:21:07 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 4230 Loading chi in step crop for list_pids : 2 ! batch 1 Loaded 189 chid ids of type : 4230 begin to crop the class : papier param for this class : {'min_score': 0.6} filtre for class : papier hashtag_id of this class : 492668766 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 52 About to insert : list_path_to_insert length 52 new photo from crops ! About to upload 52 photos upload in portfolio : 4869462 init cache_photo without model_param we have 52 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740658869_3572364 we have uploaded 52 photos in the portfolio 4869462 time of upload the photos Elapsed time : 22.54300880432129 we have finished the crop for the class : papier begin to crop the class : carton param for this class : {'min_score': 0.6} filtre for class : carton hashtag_id of this class : 492774966 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 27 About to insert : list_path_to_insert length 27 new photo from crops ! About to upload 27 photos upload in portfolio : 4869462 init cache_photo without model_param we have 27 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740658892_3572364 we have uploaded 27 photos in the portfolio 4869462 time of upload the photos Elapsed time : 6.777019023895264 we have finished the crop for the class : carton begin to crop the class : metal param for this class : {'min_score': 0.6} filtre for class : metal hashtag_id of this class : 492628673 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740658899_3572364 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.1696488857269287 we have finished the crop for the class : metal begin to crop the class : pet_clair param for this class : {'min_score': 0.6} filtre for class : pet_clair hashtag_id of this class : 2107755846 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! About to upload 10 photos upload in portfolio : 4869462 init cache_photo without model_param we have 10 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740658900_3572364 we have uploaded 10 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.8267600536346436 we have finished the crop for the class : pet_clair begin to crop the class : autre param for this class : {'min_score': 0.6} filtre for class : autre hashtag_id of this class : 494826614 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 31 About to insert : list_path_to_insert length 31 new photo from crops ! About to upload 31 photos upload in portfolio : 4869462 init cache_photo without model_param we have 31 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740658904_3572364 we have uploaded 31 photos in the portfolio 4869462 time of upload the photos Elapsed time : 10.744507074356079 we have finished the crop for the class : autre begin to crop the class : pehd param for this class : {'min_score': 0.6} filtre for class : pehd hashtag_id of this class : 628944319 begin to crop the class : pet_fonce param for this class : {'min_score': 0.6} filtre for class : pet_fonce hashtag_id of this class : 2107755900 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1339905620, 1339905603] Looping around the photos to save general results len do output : 124 /1339913210Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913211Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913212Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913213Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913214Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913215Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913216Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913217Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913218Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913219Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913220Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913221Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913222Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913223Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913224Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913225Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913226Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913227Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913228Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913229Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913230Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913231Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913232Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913233Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913234Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913235Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913236Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913237Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913238Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913239Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913240Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913241Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913242Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913243Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913244Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913245Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913246Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913247Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913248Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913249Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913250Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913251Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913252Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913253Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913254Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913256Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913257Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913258Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913259Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913260Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913261Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913262Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913270Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913271Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913272Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913273Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913275Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913277Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913278Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913279Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913280Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913281Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913282Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913283Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913284Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913285Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913286Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913287Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913288Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913289Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913290Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913291Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913292Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913293Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913294Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913295Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913296Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913297Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913298Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913301Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913302Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913303Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913304Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913307Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913308Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913310Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913311Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913312Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913313Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913315Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913316Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913317Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913318Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913329Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913330Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913331Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913332Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913333Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913334Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913335Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913336Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913337Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913338Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913339Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913340Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913341Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913342Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913343Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913344Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913345Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913346Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913347Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913348Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913350Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913351Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913353Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913354Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913355Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913356Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913357Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913358Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913359Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913360Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339913361Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905620', None, None, None, None, None, '2619667') ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905603', None, None, None, None, None, '2619667') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 374 time used for this insertion : 0.026393413543701172 save_final save missing photos in datou_result : time spend for datou_step_exec : 46.90207242965698 time spend to save output : 0.029877662658691406 total time spend for step 5 : 46.931950092315674 step6:thcl Thu Feb 27 13:21:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! we are using the classfication for only one thcl 3237 time to import caffe and check if the image exist : 0.009625434875488281 time to convert the images to numpy array : 0.012906074523925781 time to import caffe and check if the image exist : 0.00558018684387207 time to convert the images to numpy array : 0.02048802375793457 time to import caffe and check if the image exist : 0.015913724899291992 time to convert the images to numpy array : 0.014034032821655273 time to import caffe and check if the image exist : 0.01198577880859375 time to convert the images to numpy array : 0.02145218849182129 time to import caffe and check if the image exist : 0.011007070541381836 time to convert the images to numpy array : 0.025191068649291992 time to import caffe and check if the image exist : 0.014122724533081055 time to convert the images to numpy array : 0.025371074676513672 time to import caffe and check if the image exist : 0.01215219497680664 time to convert the images to numpy array : 0.03238987922668457 time to import caffe and check if the image exist : 0.016153812408447266 time to convert the images to numpy array : 0.03046870231628418 time to import caffe and check if the image exist : 0.04847240447998047 time to convert the images to numpy array : 0.004777669906616211 time to import caffe and check if the image exist : 0.007592916488647461 time to convert the images to numpy array : 0.0448455810546875 total time to convert the images to numpy array : 0.2883718013763428 list photo_ids error: [] list photo_ids correct : [1339913249, 1339913250, 1339913251, 1339913252, 1339913253, 1339913254, 1339913256, 1339913257, 1339913258, 1339913259, 1339913260, 1339913261, 1339913262, 1339913210, 1339913211, 1339913212, 1339913213, 1339913214, 1339913215, 1339913216, 1339913217, 1339913218, 1339913219, 1339913220, 1339913221, 1339913222, 1339913270, 1339913271, 1339913272, 1339913273, 1339913275, 1339913277, 1339913278, 1339913279, 1339913280, 1339913281, 1339913282, 1339913283, 1339913284, 1339913317, 1339913318, 1339913329, 1339913330, 1339913331, 1339913332, 1339913333, 1339913334, 1339913335, 1339913336, 1339913337, 1339913338, 1339913339, 1339913340, 1339913341, 1339913342, 1339913343, 1339913344, 1339913345, 1339913346, 1339913347, 1339913348, 1339913350, 1339913351, 1339913353, 1339913354, 1339913298, 1339913301, 1339913302, 1339913303, 1339913304, 1339913307, 1339913308, 1339913310, 1339913311, 1339913312, 1339913313, 1339913315, 1339913316, 1339913236, 1339913237, 1339913238, 1339913239, 1339913240, 1339913241, 1339913242, 1339913243, 1339913244, 1339913245, 1339913246, 1339913247, 1339913248, 1339913285, 1339913286, 1339913287, 1339913288, 1339913289, 1339913290, 1339913291, 1339913292, 1339913293, 1339913294, 1339913295, 1339913296, 1339913297, 1339913223, 1339913224, 1339913225, 1339913226, 1339913227, 1339913228, 1339913229, 1339913230, 1339913231, 1339913232, 1339913233, 1339913234, 1339913235, 1339913355, 1339913356, 1339913357, 1339913358, 1339913359, 1339913360, 1339913361] number of photos to traite : 124 try to delete the photos incorrect in DB tagging for thcl : 3237 To do loadFromThcl(), then load ParamDescType : thcl3237 thcls : [{'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'}] thcl {'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'} Update svm_hashtag_type_desc : 5561 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) To loadFromThcl() : net_5561 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10591 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_rubbia_refus_2500 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_rubbia_refus_2500 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_rubbia_refus_2500 /data/models_weight/learn_rubbia_refus_2500/caffemodel size_local : 94358479 size in s3 : 94358479 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:16 caffemodel already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt size_local : 32544 size in s3 : 32544 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:15 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:09:52 mean.npy already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/synset_words.txt size_local : 334 size in s3 : 334 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:10:06 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt caffemodel_filename : /data/models_weight/learn_rubbia_refus_2500/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10372 max_wait_temp : 1 max_wait : 0 dict_keys(['prob', 'pool5']) time used to do the prepocess of the images : 0.5890448093414307 time used to do the prediction : 0.5170760154724121 save descriptor for thcl : 3237 time to traite the descriptors : 0.78173828125 Catched exception ! Connect or reconnect ! storage_type for insertDescriptorsMulti : 3 To insert : 1339913249 To insert : 1339913250 To insert : 1339913251 To insert : 1339913252 To insert : 1339913253 To insert : 1339913254 To insert : 1339913256 To insert : 1339913257 To insert : 1339913258 To insert : 1339913259 To insert : 1339913260 To insert : 1339913261 To insert : 1339913262 To insert : 1339913210 To insert : 1339913211 To insert : 1339913212 To insert : 1339913213 To insert : 1339913214 To insert : 1339913215 To insert : 1339913216 To insert : 1339913217 To insert : 1339913218 To insert : 1339913219 To insert : 1339913220 To insert : 1339913221 To insert : 1339913222 To insert : 1339913270 To insert : 1339913271 To insert : 1339913272 To insert : 1339913273 To insert : 1339913275 To insert : 1339913277 To insert : 1339913278 To insert : 1339913279 To insert : 1339913280 To insert : 1339913281 To insert : 1339913282 To insert : 1339913283 To insert : 1339913284 To insert : 1339913317 To insert : 1339913318 To insert : 1339913329 To insert : 1339913330 To insert : 1339913331 To insert : 1339913332 To insert : 1339913333 To insert : 1339913334 To insert : 1339913335 To insert : 1339913336 To insert : 1339913337 To insert : 1339913338 To insert : 1339913339 To insert : 1339913340 To insert : 1339913341 To insert : 1339913342 To insert : 1339913343 To insert : 1339913344 To insert : 1339913345 To insert : 1339913346 To insert : 1339913347 To insert : 1339913348 To insert : 1339913350 To insert : 1339913351 To insert : 1339913353 To insert : 1339913354 To insert : 1339913298 To insert : 1339913301 To insert : 1339913302 To insert : 1339913303 To insert : 1339913304 To insert : 1339913307 To insert : 1339913308 To insert : 1339913310 To insert : 1339913311 To insert : 1339913312 To insert : 1339913313 To insert : 1339913315 To insert : 1339913316 To insert : 1339913236 To insert : 1339913237 To insert : 1339913238 To insert : 1339913239 To insert : 1339913240 To insert : 1339913241 To insert : 1339913242 To insert : 1339913243 To insert : 1339913244 To insert : 1339913245 To insert : 1339913246 To insert : 1339913247 To insert : 1339913248 To insert : 1339913285 To insert : 1339913286 To insert : 1339913287 To insert : 1339913288 To insert : 1339913289 To insert : 1339913290 To insert : 1339913291 To insert : 1339913292 To insert : 1339913293 To insert : 1339913294 To insert : 1339913295 To insert : 1339913296 To insert : 1339913297 To insert : 1339913223 To insert : 1339913224 To insert : 1339913225 To insert : 1339913226 To insert : 1339913227 To insert : 1339913228 To insert : 1339913229 To insert : 1339913230 To insert : 1339913231 To insert : 1339913232 To insert : 1339913233 To insert : 1339913234 To insert : 1339913235 To insert : 1339913355 To insert : 1339913356 To insert : 1339913357 To insert : 1339913358 To insert : 1339913359 To insert : 1339913360 To insert : 1339913361 time to insert the descriptors : 28.523799419403076 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos SAVE THCL : begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 200 time used for this insertion : 0.02277064323425293 save missing photos in datou_result : time spend for datou_step_exec : 34.751413106918335 time spend to save output : 0.06863665580749512 total time spend for step 6 : 34.82004976272583 step7:ventilate_hashtags_in_portfolio Thu Feb 27 13:22:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! Iterating over portfolio : 20881153 get user id for portfolio 20881153 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20881153 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton','environnement','pehd','Film_plastique','metal','papier','PET_fonce','Papier','pet_clair','flou','pet_fonce','carton','mal_croppe','refus','autre','PEHD','Tetrapak','PET_clair')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20881153 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton','environnement','pehd','Film_plastique','metal','papier','PET_fonce','Papier','pet_clair','flou','pet_fonce','carton','mal_croppe','refus','autre','PEHD','Tetrapak','PET_clair')) AND mptpi.`min_score`=0.6 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20881153 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton','environnement','pehd','Film_plastique','metal','papier','PET_fonce','Papier','pet_clair','flou','pet_fonce','carton','mal_croppe','refus','autre','PEHD','Tetrapak','PET_clair')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20881153 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton','environnement','pehd','Film_plastique','metal','papier','PET_fonce','Papier','pet_clair','flou','pet_fonce','carton','mal_croppe','refus','autre','PEHD','Tetrapak','PET_clair')) AND mptpi.`min_score`=0.6 To do lien utilise dans velours : https://www.fotonower.com/velours/20881434,20881424,20881438,20881426,20881427,20881430,20881433,20881440,20881432,20881435,20881436,20881437,20881439?tags=carton,environnement,pehd,film_plastique,metal,papier,pet_fonce,pet_clair,flou,mal_croppe,refus,autre,tetrapak&datou_id_consolidate=4235&port_consolidate=20881153 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1339905620, 1339905603] Looping around the photos to save general results len do output : 1 /20881153. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905620', None, None, None, None, None, '2619667') ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905603', None, None, None, None, None, '2619667') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.012691497802734375 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.740804672241211 time spend to save output : 0.012929201126098633 total time spend for step 7 : 2.7537338733673096 step8:final Thu Feb 27 13:22:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1339905620: ('0.2407666415285407',), 1339905603: ('0.2407666415285407',)} new output for save of step final : {1339905620: ('0.2407666415285407',), 1339905603: ('0.2407666415285407',)} [1339905620, 1339905603] Looping around the photos to save general results len do output : 2 /1339905620.Didn't retrieve data . /1339905603.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905620', None, None, None, None, None, '2619667') ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905603', None, None, None, None, None, '2619667') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.01283121109008789 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.12081146240234375 time spend to save output : 0.013037919998168945 total time spend for step 8 : 0.1338493824005127 step9:velours_tree Thu Feb 27 13:22:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure list_portfolios : 20881434,20881424,20881438,20881426,20881427,20881430,20881433,20881440,20881432,20881435,20881436,20881437,20881439 photo desc type : 5561 - Retrieving photos to tag... query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881434 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881424 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881438 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881426 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881427 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881430 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881433 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881440 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881432 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881435 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881436 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881437 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881439 ORDER BY ph.size desc - Loading descriptors... Size : 2048 len(descriptors) : 124 Compute structured hierarchical clustering... ward : AgglomerativeClustering(n_clusters=6) ward.labels_ : [1 2 1 2 3 1 0 0 0 1 0 1 2 1 1 3 1 1 0 1 1 3 5 0 2 3 3 2 2 2 3 0 0 0 3 3 3 0 0 1 0 3 3 1 0 1 3 0 1 2 0 3 4 4 0 4 3 0 0 3 0 2 2 2 1 0 4 4 0 3 0 1 3 0 3 2 2 1 0 3 3 3 1 2 1 1 0 0 0 2 1 3 1 5 0 0 0 3 3 1 1 1 4 3 1 1 1 4 5 3 0 0 3 0 0 0 3 1 1 3 4 1 1 1] Elapsed time: 0.019984006881713867 graph_id used : 80517 - Beta version, working pretty good on 11-5-16 ! https://www.fotonower.com/velours/20881434,20881424,20881438,20881426,20881427,20881430,20881433,20881440,20881432,20881435,20881436,20881437,20881439?tags=carton,environnement,pehd,film_plastique,metal,papier,pet_fonce,pet_clair,flou,mal_croppe,refus,autre,tetrapak&datou_id_consolidate=4235&port_consolidate=20881153&tree_id=80517 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : velours_tree we use saveGeneral [1339905620, 1339905603] Looping around the photos to save general results len do output : 1 /20881153Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905620', None, None, None, None, None, '2619667') ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905603', None, None, None, None, None, '2619667') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.012376070022583008 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.860286474227905 time spend to save output : 0.012715578079223633 total time spend for step 9 : 7.873002052307129 step10:send_mail_cod Thu Feb 27 13:22:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Qualipapia_P20881153_27-02-2025_13_22_40.pdf 20881416 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208814161740658960 20881420 imagette208814201740658961 20881408 imagette208814081740658961 20881409 change filename to text .change filename to text .change filename to text .change filename to text .imagette208814091740658961 20881412 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208814121740658961 20881415 imagette208814151740658962 20881422 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208814221740658962 20881414 imagette208814141740658962 20881417 imagette208814171740658962 20881418 imagette208814181740658962 20881419 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208814191740658962 20881421 imagette208814211740658963 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20881153 and hashtag_type = 4230 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20881434,20881424,20881438,20881426,20881427,20881430,20881433,20881440,20881432,20881435,20881436,20881437,20881439?tags=carton,environnement,pehd,film_plastique,metal,papier,pet_fonce,pet_clair,flou,mal_croppe,refus,autre,tetrapak&datou_id_consolidate=4235&port_consolidate=20881153&tree_id=80517 your option no_mail is active, we will not send the real mail to your client args[1339905620] : ((1339905620, -7.6054192957272955, 492609224), (1339905620, -0.32349591817548906, 496442774), '0.2407666415285407') We are sending mail with results at cod@fotonower.com args[1339905603] : ((1339905603, -7.594240120299947, 492609224), (1339905603, -0.3163410595036382, 496442774), '0.2407666415285407') We are sending mail with results at cod@fotonower.com refus_total : 0.2407666415285407 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20881153 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20881153_27-02-2025_13_22_40.pdf results_Qualipapia_P20881153_27-02-2025_13_22_40.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20881153_27-02-2025_13_22_40.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('4234','20881153','results_Qualipapia_P20881153_27-02-2025_13_22_40.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20881153_27-02-2025_13_22_40.pdf','pdf','','0.16','0.2407666415285407') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1339905620, 1339905603] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905620', None, None, None, None, None, '2619667') ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905603', None, None, None, None, None, '2619667') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 2 time used for this insertion : 0.011632204055786133 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.494251251220703 time spend to save output : 0.011852502822875977 total time spend for step 10 : 4.506103754043579 step11:split_time_score Thu Feb 27 13:22:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3442, 'mtr_user_id': 31, 'name': 'classifieur_2camions_valcor_021122_v1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'deux_camions,camion_droite,camion_gauche,pas_de_camion', 'svm_portfolios_learning': '7659379,7659034,7657685,7657114', 'photo_hashtag_type': 4458, 'photo_desc_type': 5723, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107760533,2107760534,2107760535,2107760536'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('09', 2),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 27022025 20881153 Nombre de photos uploadées : 2 / 23040 (0%) 27022025 20881153 Nombre de photos taguées (types de déchets): 2 / 2 (100%) 27022025 20881153 Nombre de photos taguées (volume) : 0 / 2 (0%) elapsed_time : load_data_split_time_score 3.5762786865234375e-06 elapsed_time : order_list_meta_photo_and_scores 8.58306884765625e-06 ?? elapsed_time : fill_and_build_computed_from_old_data 0.0002818107604980469 elapsed_time : insert_dashboard_record_day_entry 0.02354264259338379 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20880914 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20880915 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20880916 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20880917 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20880918 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20880919 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20880920 order by id desc limit 1 Qualite : 0.19961527896716455 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20880921_27-02-2025_12_49_38.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20880921 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20880921 AND mptpi.`type`=4230 To do Qualite : 0.2407666415285407 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20881153_27-02-2025_13_22_40.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20881153 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20881153 AND mptpi.`type`=4230 To do Qualite : 0.16957394556367378 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20881154_27-02-2025_13_19_31.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20881154 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20881154 AND mptpi.`type`=4230 To do Qualite : 0.1847201060328798 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20881155 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20881155 AND mptpi.`type`=4230 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'27022025': {'nb_upload': 2, 'nb_taggue_class': 2, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1339905620, 1339905603] Looping around the photos to save general results len do output : 1 /20881153Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905620', None, None, None, None, None, '2619667') ('4234', None, None, None, None, None, None, None, '2619667') ('4234', '20881153', '1339905603', None, None, None, None, None, '2619667') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.012576580047607422 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.596659421920776 time spend to save output : 0.012779951095581055 total time spend for step 11 : 6.609439373016357 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 1 len(input) + len(total_photo_id_missing) : 2 set_done_treatment 34.34user 22.46system 2:26.08elapsed 38%CPU (0avgtext+0avgdata 2501772maxresident)k 779312inputs+11216outputs (616major+2143995minor)pagefaults 0swaps