python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3550052 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 28>10 please execute split_portfolio.py -i 20254669 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2551093'] with mtr_portfolio_ids : ['20254669'] and first list_photo_ids : [] new path : /proc/3550052/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.6958858966827393 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Wed Feb 5 12:00:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10770 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-05 12:00:33.071674: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-05 12:00:33.099234: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-05 12:00:33.101523: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f09c4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-05 12:00:33.101559: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-05 12:00:33.105869: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-05 12:00:33.241723: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1cf2a220 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-05 12:00:33.241784: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-05 12:00:33.243394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 12:00:33.243844: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 12:00:33.247071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 12:00:33.250260: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 12:00:33.250791: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 12:00:33.254321: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 12:00:33.256163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 12:00:33.261065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 12:00:33.262481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 12:00:33.262557: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 12:00:33.263348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-05 12:00:33.263365: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-05 12:00:33.263375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-05 12:00:33.265092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9259 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-05 12:00:33.534965: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 12:00:33.535092: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 12:00:33.535125: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 12:00:33.535152: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 12:00:33.535178: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 12:00:33.535203: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 12:00:33.535228: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 12:00:33.535254: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 12:00:33.536634: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 12:00:33.537843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 12:00:33.537893: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 12:00:33.537909: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 12:00:33.537923: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 12:00:33.537936: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 12:00:33.537950: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 12:00:33.537964: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 12:00:33.537978: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 12:00:33.539235: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 12:00:33.539267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-05 12:00:33.539277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-05 12:00:33.539285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-05 12:00:33.540577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9259 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-05 12:00:41.671976: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 12:00:41.853283: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 24 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 21 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 26 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 18 Detection mask done ! Trying to reset tf kernel 3553442 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3234 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 7062 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0536041259765625 nb_pixel_total : 3054429 time to create 1 rle with new method : 0.13647937774658203 length of segment : 3237 time for calcul the mask position with numpy : 0.0008363723754882812 nb_pixel_total : 31916 time to create 1 rle with old method : 0.03483724594116211 length of segment : 334 time for calcul the mask position with numpy : 0.00011610984802246094 nb_pixel_total : 2648 time to create 1 rle with old method : 0.0031626224517822266 length of segment : 58 time for calcul the mask position with numpy : 0.004011392593383789 nb_pixel_total : 285083 time to create 1 rle with new method : 0.01371622085571289 length of segment : 454 time for calcul the mask position with numpy : 0.0034508705139160156 nb_pixel_total : 248589 time to create 1 rle with new method : 0.016911029815673828 length of segment : 481 time for calcul the mask position with numpy : 0.0028231143951416016 nb_pixel_total : 124932 time to create 1 rle with old method : 0.13465094566345215 length of segment : 645 time for calcul the mask position with numpy : 0.00014209747314453125 nb_pixel_total : 2795 time to create 1 rle with old method : 0.003450155258178711 length of segment : 46 time for calcul the mask position with numpy : 0.0002155303955078125 nb_pixel_total : 5882 time to create 1 rle with old method : 0.0070803165435791016 length of segment : 144 time for calcul the mask position with numpy : 0.0002713203430175781 nb_pixel_total : 3877 time to create 1 rle with old method : 0.0052394866943359375 length of segment : 112 time for calcul the mask position with numpy : 0.0039522647857666016 nb_pixel_total : 215851 time to create 1 rle with new method : 0.011235475540161133 length of segment : 743 time for calcul the mask position with numpy : 0.0026221275329589844 nb_pixel_total : 135247 time to create 1 rle with old method : 0.14795804023742676 length of segment : 392 time for calcul the mask position with numpy : 0.0046498775482177734 nb_pixel_total : 240575 time to create 1 rle with new method : 0.013268470764160156 length of segment : 499 time for calcul the mask position with numpy : 0.0003025531768798828 nb_pixel_total : 6479 time to create 1 rle with old method : 0.00767064094543457 length of segment : 101 time for calcul the mask position with numpy : 0.0016984939575195312 nb_pixel_total : 58964 time to create 1 rle with old method : 0.06482982635498047 length of segment : 322 time for calcul the mask position with numpy : 0.0021257400512695312 nb_pixel_total : 112877 time to create 1 rle with old method : 0.12338566780090332 length of segment : 458 time for calcul the mask position with numpy : 0.0018410682678222656 nb_pixel_total : 94078 time to create 1 rle with old method : 0.1074984073638916 length of segment : 463 time for calcul the mask position with numpy : 0.0018012523651123047 nb_pixel_total : 65140 time to create 1 rle with old method : 0.10486626625061035 length of segment : 315 time for calcul the mask position with numpy : 0.00013375282287597656 nb_pixel_total : 974 time to create 1 rle with old method : 0.0012345314025878906 length of segment : 44 time for calcul the mask position with numpy : 8.797645568847656e-05 nb_pixel_total : 1174 time to create 1 rle with old method : 0.0014431476593017578 length of segment : 49 time for calcul the mask position with numpy : 0.0007481575012207031 nb_pixel_total : 27550 time to create 1 rle with old method : 0.030894756317138672 length of segment : 266 time for calcul the mask position with numpy : 0.001990079879760742 nb_pixel_total : 101231 time to create 1 rle with old method : 0.11506366729736328 length of segment : 244 time for calcul the mask position with numpy : 0.0013151168823242188 nb_pixel_total : 54420 time to create 1 rle with old method : 0.06238818168640137 length of segment : 338 time for calcul the mask position with numpy : 0.00017261505126953125 nb_pixel_total : 5340 time to create 1 rle with old method : 0.006124258041381836 length of segment : 113 time for calcul the mask position with numpy : 0.0010483264923095703 nb_pixel_total : 51796 time to create 1 rle with old method : 0.05812859535217285 length of segment : 367 time for calcul the mask position with numpy : 0.00037598609924316406 nb_pixel_total : 15167 time to create 1 rle with old method : 0.01680922508239746 length of segment : 167 time for calcul the mask position with numpy : 0.00037550926208496094 nb_pixel_total : 20164 time to create 1 rle with old method : 0.022849559783935547 length of segment : 157 time for calcul the mask position with numpy : 0.0021619796752929688 nb_pixel_total : 93910 time to create 1 rle with old method : 0.10093879699707031 length of segment : 461 time for calcul the mask position with numpy : 0.0016560554504394531 nb_pixel_total : 82379 time to create 1 rle with old method : 0.09168648719787598 length of segment : 664 time for calcul the mask position with numpy : 0.0033385753631591797 nb_pixel_total : 129838 time to create 1 rle with old method : 0.14437222480773926 length of segment : 590 time for calcul the mask position with numpy : 0.0010890960693359375 nb_pixel_total : 44337 time to create 1 rle with old method : 0.04943132400512695 length of segment : 243 time for calcul the mask position with numpy : 0.0018894672393798828 nb_pixel_total : 69355 time to create 1 rle with old method : 0.0987696647644043 length of segment : 395 time for calcul the mask position with numpy : 0.0015921592712402344 nb_pixel_total : 49088 time to create 1 rle with old method : 0.05442380905151367 length of segment : 361 time for calcul the mask position with numpy : 0.0014719963073730469 nb_pixel_total : 63085 time to create 1 rle with old method : 0.06989598274230957 length of segment : 569 time for calcul the mask position with numpy : 0.0022051334381103516 nb_pixel_total : 85849 time to create 1 rle with old method : 0.11446809768676758 length of segment : 577 time for calcul the mask position with numpy : 0.001283884048461914 nb_pixel_total : 40045 time to create 1 rle with old method : 0.04431509971618652 length of segment : 310 time for calcul the mask position with numpy : 0.0011513233184814453 nb_pixel_total : 39623 time to create 1 rle with old method : 0.044123172760009766 length of segment : 327 time for calcul the mask position with numpy : 0.00026297569274902344 nb_pixel_total : 2939 time to create 1 rle with old method : 0.0036385059356689453 length of segment : 90 time for calcul the mask position with numpy : 0.00019431114196777344 nb_pixel_total : 1601 time to create 1 rle with old method : 0.0019347667694091797 length of segment : 65 time for calcul the mask position with numpy : 0.0009448528289794922 nb_pixel_total : 25469 time to create 1 rle with old method : 0.029937267303466797 length of segment : 241 time for calcul the mask position with numpy : 0.0005776882171630859 nb_pixel_total : 23445 time to create 1 rle with old method : 0.026540040969848633 length of segment : 202 time for calcul the mask position with numpy : 0.0011506080627441406 nb_pixel_total : 44676 time to create 1 rle with old method : 0.04911017417907715 length of segment : 259 time for calcul the mask position with numpy : 0.0003628730773925781 nb_pixel_total : 11348 time to create 1 rle with old method : 0.012914657592773438 length of segment : 315 time for calcul the mask position with numpy : 0.001215219497680664 nb_pixel_total : 43658 time to create 1 rle with old method : 0.04942488670349121 length of segment : 250 time for calcul the mask position with numpy : 0.00016260147094726562 nb_pixel_total : 2131 time to create 1 rle with old method : 0.002596616744995117 length of segment : 55 time for calcul the mask position with numpy : 0.0001735687255859375 nb_pixel_total : 3113 time to create 1 rle with old method : 0.003770112991333008 length of segment : 100 time for calcul the mask position with numpy : 0.0071451663970947266 nb_pixel_total : 358550 time to create 1 rle with new method : 0.02079033851623535 length of segment : 757 time for calcul the mask position with numpy : 0.004485607147216797 nb_pixel_total : 147002 time to create 1 rle with old method : 0.1662428379058838 length of segment : 556 time for calcul the mask position with numpy : 0.0006172657012939453 nb_pixel_total : 10102 time to create 1 rle with old method : 0.01163482666015625 length of segment : 163 time for calcul the mask position with numpy : 0.002376556396484375 nb_pixel_total : 106575 time to create 1 rle with old method : 0.13825607299804688 length of segment : 432 time for calcul the mask position with numpy : 0.0008397102355957031 nb_pixel_total : 35056 time to create 1 rle with old method : 0.03869199752807617 length of segment : 341 time for calcul the mask position with numpy : 0.0023882389068603516 nb_pixel_total : 82946 time to create 1 rle with old method : 0.09488153457641602 length of segment : 598 time for calcul the mask position with numpy : 0.001932382583618164 nb_pixel_total : 45863 time to create 1 rle with old method : 0.05055856704711914 length of segment : 316 time for calcul the mask position with numpy : 0.0003352165222167969 nb_pixel_total : 2385 time to create 1 rle with old method : 0.002846240997314453 length of segment : 86 time for calcul the mask position with numpy : 0.00048613548278808594 nb_pixel_total : 3820 time to create 1 rle with old method : 0.0045812129974365234 length of segment : 178 time for calcul the mask position with numpy : 0.0003590583801269531 nb_pixel_total : 6553 time to create 1 rle with old method : 0.0077130794525146484 length of segment : 118 time for calcul the mask position with numpy : 0.002012491226196289 nb_pixel_total : 34308 time to create 1 rle with old method : 0.03885483741760254 length of segment : 365 time for calcul the mask position with numpy : 0.004010438919067383 nb_pixel_total : 123180 time to create 1 rle with old method : 0.14281892776489258 length of segment : 370 time for calcul the mask position with numpy : 0.004422426223754883 nb_pixel_total : 78494 time to create 1 rle with old method : 0.11139726638793945 length of segment : 426 time for calcul the mask position with numpy : 0.0004200935363769531 nb_pixel_total : 6208 time to create 1 rle with old method : 0.0074615478515625 length of segment : 79 time for calcul the mask position with numpy : 0.00014448165893554688 nb_pixel_total : 1004 time to create 1 rle with old method : 0.0013132095336914062 length of segment : 41 time for calcul the mask position with numpy : 0.0002422332763671875 nb_pixel_total : 2657 time to create 1 rle with old method : 0.0034122467041015625 length of segment : 57 time for calcul the mask position with numpy : 0.0019030570983886719 nb_pixel_total : 64931 time to create 1 rle with old method : 0.07341289520263672 length of segment : 454 time for calcul the mask position with numpy : 0.0019321441650390625 nb_pixel_total : 57396 time to create 1 rle with old method : 0.06359314918518066 length of segment : 394 time for calcul the mask position with numpy : 0.07392263412475586 nb_pixel_total : 2798055 time to create 1 rle with new method : 0.37099123001098633 length of segment : 3665 time for calcul the mask position with numpy : 0.0002551078796386719 nb_pixel_total : 13223 time to create 1 rle with old method : 0.014950752258300781 length of segment : 152 time for calcul the mask position with numpy : 0.00010609626770019531 nb_pixel_total : 2784 time to create 1 rle with old method : 0.003452777862548828 length of segment : 48 time for calcul the mask position with numpy : 0.005962848663330078 nb_pixel_total : 351987 time to create 1 rle with new method : 0.014913558959960938 length of segment : 725 time for calcul the mask position with numpy : 0.005544185638427734 nb_pixel_total : 386279 time to create 1 rle with new method : 0.01320648193359375 length of segment : 713 time for calcul the mask position with numpy : 0.0017178058624267578 nb_pixel_total : 115209 time to create 1 rle with old method : 0.12640857696533203 length of segment : 585 time for calcul the mask position with numpy : 0.000606536865234375 nb_pixel_total : 19767 time to create 1 rle with old method : 0.022732019424438477 length of segment : 170 time for calcul the mask position with numpy : 0.000244140625 nb_pixel_total : 2830 time to create 1 rle with old method : 0.0033562183380126953 length of segment : 73 time for calcul the mask position with numpy : 0.0010406970977783203 nb_pixel_total : 15649 time to create 1 rle with old method : 0.017724275588989258 length of segment : 264 time for calcul the mask position with numpy : 0.0006186962127685547 nb_pixel_total : 10074 time to create 1 rle with old method : 0.016846418380737305 length of segment : 134 time for calcul the mask position with numpy : 0.0003418922424316406 nb_pixel_total : 4426 time to create 1 rle with old method : 0.007485866546630859 length of segment : 126 time for calcul the mask position with numpy : 0.012224435806274414 nb_pixel_total : 389575 time to create 1 rle with new method : 0.015184640884399414 length of segment : 749 time for calcul the mask position with numpy : 0.044889211654663086 nb_pixel_total : 1457353 time to create 1 rle with new method : 0.07728838920593262 length of segment : 2381 time for calcul the mask position with numpy : 0.0023293495178222656 nb_pixel_total : 70106 time to create 1 rle with old method : 0.07956647872924805 length of segment : 272 time for calcul the mask position with numpy : 0.0009429454803466797 nb_pixel_total : 31881 time to create 1 rle with old method : 0.04215717315673828 length of segment : 214 time for calcul the mask position with numpy : 0.0005214214324951172 nb_pixel_total : 23429 time to create 1 rle with old method : 0.025823116302490234 length of segment : 210 time for calcul the mask position with numpy : 0.0004177093505859375 nb_pixel_total : 12137 time to create 1 rle with old method : 0.013991594314575195 length of segment : 79 time for calcul the mask position with numpy : 0.0005755424499511719 nb_pixel_total : 29773 time to create 1 rle with old method : 0.03641772270202637 length of segment : 232 time for calcul the mask position with numpy : 0.00018930435180664062 nb_pixel_total : 8242 time to create 1 rle with old method : 0.009299039840698242 length of segment : 109 time for calcul the mask position with numpy : 0.007841825485229492 nb_pixel_total : 494559 time to create 1 rle with new method : 0.013669013977050781 length of segment : 1075 time for calcul the mask position with numpy : 0.000789642333984375 nb_pixel_total : 27534 time to create 1 rle with old method : 0.036923885345458984 length of segment : 264 time for calcul the mask position with numpy : 0.002743959426879883 nb_pixel_total : 90318 time to create 1 rle with old method : 0.12546086311340332 length of segment : 350 time for calcul the mask position with numpy : 0.0007021427154541016 nb_pixel_total : 15863 time to create 1 rle with old method : 0.01829218864440918 length of segment : 226 time for calcul the mask position with numpy : 0.00018310546875 nb_pixel_total : 1566 time to create 1 rle with old method : 0.001972198486328125 length of segment : 77 time for calcul the mask position with numpy : 0.026691675186157227 nb_pixel_total : 874697 time to create 1 rle with new method : 0.040344953536987305 length of segment : 1667 time for calcul the mask position with numpy : 0.001402139663696289 nb_pixel_total : 47108 time to create 1 rle with old method : 0.05323481559753418 length of segment : 226 time for calcul the mask position with numpy : 0.0052797794342041016 nb_pixel_total : 219927 time to create 1 rle with new method : 0.00813150405883789 length of segment : 567 time for calcul the mask position with numpy : 0.0003571510314941406 nb_pixel_total : 5304 time to create 1 rle with old method : 0.006429195404052734 length of segment : 101 time for calcul the mask position with numpy : 0.013249635696411133 nb_pixel_total : 522657 time to create 1 rle with new method : 0.023565053939819336 length of segment : 959 time for calcul the mask position with numpy : 0.002021312713623047 nb_pixel_total : 150692 time to create 1 rle with new method : 0.003990650177001953 length of segment : 415 time for calcul the mask position with numpy : 0.0007984638214111328 nb_pixel_total : 22952 time to create 1 rle with old method : 0.02624058723449707 length of segment : 213 time for calcul the mask position with numpy : 0.0004355907440185547 nb_pixel_total : 22839 time to create 1 rle with old method : 0.026073217391967773 length of segment : 210 time for calcul the mask position with numpy : 0.011155843734741211 nb_pixel_total : 348599 time to create 1 rle with new method : 0.022054433822631836 length of segment : 890 time for calcul the mask position with numpy : 0.0005831718444824219 nb_pixel_total : 22637 time to create 1 rle with old method : 0.03432011604309082 length of segment : 192 time for calcul the mask position with numpy : 0.0011038780212402344 nb_pixel_total : 18389 time to create 1 rle with old method : 0.02697277069091797 length of segment : 223 time for calcul the mask position with numpy : 0.002065896987915039 nb_pixel_total : 76855 time to create 1 rle with old method : 0.08273768424987793 length of segment : 336 time for calcul the mask position with numpy : 0.0017142295837402344 nb_pixel_total : 33696 time to create 1 rle with old method : 0.03736162185668945 length of segment : 264 time for calcul the mask position with numpy : 0.0014319419860839844 nb_pixel_total : 37426 time to create 1 rle with old method : 0.04170417785644531 length of segment : 347 time for calcul the mask position with numpy : 0.001201629638671875 nb_pixel_total : 30218 time to create 1 rle with old method : 0.0340728759765625 length of segment : 243 time for calcul the mask position with numpy : 0.0010123252868652344 nb_pixel_total : 23916 time to create 1 rle with old method : 0.03004741668701172 length of segment : 180 time for calcul the mask position with numpy : 0.0036652088165283203 nb_pixel_total : 96168 time to create 1 rle with old method : 0.10872650146484375 length of segment : 449 time for calcul the mask position with numpy : 0.001111745834350586 nb_pixel_total : 33772 time to create 1 rle with old method : 0.03804421424865723 length of segment : 234 time for calcul the mask position with numpy : 0.0011336803436279297 nb_pixel_total : 26506 time to create 1 rle with old method : 0.030992984771728516 length of segment : 379 time for calcul the mask position with numpy : 0.0006008148193359375 nb_pixel_total : 12746 time to create 1 rle with old method : 0.014258146286010742 length of segment : 194 time for calcul the mask position with numpy : 0.001924753189086914 nb_pixel_total : 16108 time to create 1 rle with old method : 0.019405603408813477 length of segment : 336 time for calcul the mask position with numpy : 0.0014073848724365234 nb_pixel_total : 36976 time to create 1 rle with old method : 0.0409848690032959 length of segment : 460 time for calcul the mask position with numpy : 0.003906965255737305 nb_pixel_total : 109171 time to create 1 rle with old method : 0.13865947723388672 length of segment : 348 time for calcul the mask position with numpy : 0.001317739486694336 nb_pixel_total : 13886 time to create 1 rle with old method : 0.019689321517944336 length of segment : 178 time for calcul the mask position with numpy : 0.008515596389770508 nb_pixel_total : 257916 time to create 1 rle with new method : 0.01306009292602539 length of segment : 689 time for calcul the mask position with numpy : 0.00640106201171875 nb_pixel_total : 173889 time to create 1 rle with new method : 0.009765148162841797 length of segment : 407 time for calcul the mask position with numpy : 0.001226186752319336 nb_pixel_total : 36317 time to create 1 rle with old method : 0.04247760772705078 length of segment : 288 time for calcul the mask position with numpy : 0.0007891654968261719 nb_pixel_total : 15940 time to create 1 rle with old method : 0.026250600814819336 length of segment : 137 time for calcul the mask position with numpy : 0.00024390220642089844 nb_pixel_total : 1720 time to create 1 rle with old method : 0.003373861312866211 length of segment : 44 time for calcul the mask position with numpy : 0.0007433891296386719 nb_pixel_total : 8176 time to create 1 rle with old method : 0.01125335693359375 length of segment : 188 time for calcul the mask position with numpy : 0.0011856555938720703 nb_pixel_total : 25481 time to create 1 rle with old method : 0.028862714767456055 length of segment : 219 time for calcul the mask position with numpy : 0.004399538040161133 nb_pixel_total : 147460 time to create 1 rle with old method : 0.16048216819763184 length of segment : 511 time for calcul the mask position with numpy : 0.009548425674438477 nb_pixel_total : 160012 time to create 1 rle with new method : 0.01684880256652832 length of segment : 701 time for calcul the mask position with numpy : 0.003359079360961914 nb_pixel_total : 94946 time to create 1 rle with old method : 0.11944127082824707 length of segment : 494 time for calcul the mask position with numpy : 0.0016202926635742188 nb_pixel_total : 50241 time to create 1 rle with old method : 0.05658388137817383 length of segment : 296 time for calcul the mask position with numpy : 0.010213851928710938 nb_pixel_total : 373604 time to create 1 rle with new method : 0.014784812927246094 length of segment : 727 time for calcul the mask position with numpy : 0.0011301040649414062 nb_pixel_total : 17823 time to create 1 rle with old method : 0.019899845123291016 length of segment : 284 time for calcul the mask position with numpy : 0.0003402233123779297 nb_pixel_total : 7896 time to create 1 rle with old method : 0.009501457214355469 length of segment : 86 time for calcul the mask position with numpy : 0.0007481575012207031 nb_pixel_total : 10517 time to create 1 rle with old method : 0.012755870819091797 length of segment : 124 time for calcul the mask position with numpy : 0.00017452239990234375 nb_pixel_total : 738 time to create 1 rle with old method : 0.0012333393096923828 length of segment : 42 time for calcul the mask position with numpy : 0.002276182174682617 nb_pixel_total : 89095 time to create 1 rle with old method : 0.11374402046203613 length of segment : 351 time for calcul the mask position with numpy : 0.0013244152069091797 nb_pixel_total : 33586 time to create 1 rle with old method : 0.0375823974609375 length of segment : 177 time for calcul the mask position with numpy : 0.0003635883331298828 nb_pixel_total : 3078 time to create 1 rle with old method : 0.003931760787963867 length of segment : 81 time for calcul the mask position with numpy : 0.0002512931823730469 nb_pixel_total : 6996 time to create 1 rle with old method : 0.00812077522277832 length of segment : 96 time for calcul the mask position with numpy : 0.00021719932556152344 nb_pixel_total : 5351 time to create 1 rle with old method : 0.006989717483520508 length of segment : 62 time for calcul the mask position with numpy : 0.0015368461608886719 nb_pixel_total : 69812 time to create 1 rle with old method : 0.11005425453186035 length of segment : 441 time for calcul the mask position with numpy : 0.002497434616088867 nb_pixel_total : 11667 time to create 1 rle with old method : 0.014073848724365234 length of segment : 304 time for calcul the mask position with numpy : 0.0001227855682373047 nb_pixel_total : 2171 time to create 1 rle with old method : 0.0029098987579345703 length of segment : 61 time for calcul the mask position with numpy : 0.008894681930541992 nb_pixel_total : 339921 time to create 1 rle with new method : 0.016096115112304688 length of segment : 583 time for calcul the mask position with numpy : 0.003453969955444336 nb_pixel_total : 109967 time to create 1 rle with old method : 0.12401604652404785 length of segment : 362 time for calcul the mask position with numpy : 0.000274658203125 nb_pixel_total : 3772 time to create 1 rle with old method : 0.004686832427978516 length of segment : 100 time for calcul the mask position with numpy : 0.0015404224395751953 nb_pixel_total : 61089 time to create 1 rle with old method : 0.07376718521118164 length of segment : 619 time for calcul the mask position with numpy : 0.0003209114074707031 nb_pixel_total : 6257 time to create 1 rle with old method : 0.007294178009033203 length of segment : 78 time for calcul the mask position with numpy : 0.00030231475830078125 nb_pixel_total : 6237 time to create 1 rle with old method : 0.00742030143737793 length of segment : 66 time for calcul the mask position with numpy : 0.0002448558807373047 nb_pixel_total : 3425 time to create 1 rle with old method : 0.004038095474243164 length of segment : 78 time for calcul the mask position with numpy : 0.00018715858459472656 nb_pixel_total : 3880 time to create 1 rle with old method : 0.004500389099121094 length of segment : 65 time for calcul the mask position with numpy : 0.0009245872497558594 nb_pixel_total : 26506 time to create 1 rle with old method : 0.029781103134155273 length of segment : 362 time for calcul the mask position with numpy : 0.00024580955505371094 nb_pixel_total : 3914 time to create 1 rle with old method : 0.004761934280395508 length of segment : 57 time for calcul the mask position with numpy : 0.00038552284240722656 nb_pixel_total : 5591 time to create 1 rle with old method : 0.006590127944946289 length of segment : 109 time for calcul the mask position with numpy : 0.00017595291137695312 nb_pixel_total : 1618 time to create 1 rle with old method : 0.0019605159759521484 length of segment : 72 time for calcul the mask position with numpy : 0.0001895427703857422 nb_pixel_total : 2891 time to create 1 rle with old method : 0.00351715087890625 length of segment : 63 time for calcul the mask position with numpy : 0.00047898292541503906 nb_pixel_total : 9497 time to create 1 rle with old method : 0.010922670364379883 length of segment : 138 time for calcul the mask position with numpy : 0.0004703998565673828 nb_pixel_total : 4135 time to create 1 rle with old method : 0.004968404769897461 length of segment : 181 time for calcul the mask position with numpy : 0.0016870498657226562 nb_pixel_total : 67557 time to create 1 rle with old method : 0.07664275169372559 length of segment : 416 time for calcul the mask position with numpy : 0.00012946128845214844 nb_pixel_total : 1014 time to create 1 rle with old method : 0.0012879371643066406 length of segment : 34 time for calcul the mask position with numpy : 0.000461578369140625 nb_pixel_total : 4468 time to create 1 rle with old method : 0.005398750305175781 length of segment : 139 time for calcul the mask position with numpy : 8.7738037109375e-05 nb_pixel_total : 1445 time to create 1 rle with old method : 0.0018589496612548828 length of segment : 44 time for calcul the mask position with numpy : 0.0002498626708984375 nb_pixel_total : 6690 time to create 1 rle with old method : 0.00804901123046875 length of segment : 127 time for calcul the mask position with numpy : 0.0008249282836914062 nb_pixel_total : 24463 time to create 1 rle with old method : 0.027208328247070312 length of segment : 176 time for calcul the mask position with numpy : 0.0015511512756347656 nb_pixel_total : 8870 time to create 1 rle with old method : 0.010315656661987305 length of segment : 241 time spent for convertir_results : 17.04241180419922 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 164 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 55778 save missing photos in datou_result : time spend for datou_step_exec : 53.14882183074951 time spend to save output : 3.221914291381836 total time spend for step 1 : 56.37073612213135 step2:crop_condition Wed Feb 5 12:01:26 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 164 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 49 About to insert : list_path_to_insert length 49 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1334711703, 1334711699, 1334711697, 1334696715, 1334696655, 1334696538, 1334513930, 1334513927, 1334497467, 1334497463] Looping around the photos to save general results len do output : 87 /-3655849740Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849761Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849772Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849767Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849771Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849766Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455177Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455202Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455222Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849731Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849748Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849760Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849777Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849778Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849806Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849815Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849747Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849742Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849803Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849810Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455170Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455201Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849730Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849732Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849728Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849738Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849741Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849743Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849737Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849754Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849765Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849755Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849759Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849753Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849762Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849758Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849756Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849757Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849776Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849773Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849769Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849782Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849774Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849783Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849784Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849795Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849794Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849789Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849788Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849798Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849811Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455162Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455171Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849814Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455176Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455185Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455180Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455172Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455181Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455183Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455193Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455209Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455194Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849817Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455197Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455195Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849816Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455196Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455207Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455230Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455224Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849796Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849799Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455165Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455178Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455173Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455174Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455179Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455204Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849736Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849752Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849775Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849805Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455161Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655849804Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455167Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455186Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711703', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711699', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711697', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696715', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696655', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696538', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513930', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513927', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497467', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497463', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 271 time used for this insertion : 0.18346190452575684 save_final save missing photos in datou_result : time spend for datou_step_exec : 40.047388315200806 time spend to save output : 0.19052362442016602 total time spend for step 2 : 40.23791193962097 step3:thcl Wed Feb 5 12:02:06 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.015497207641601562 time to convert the images to numpy array : 0.19985055923461914 time to import caffe and check if the image exist : 0.018802642822265625 time to convert the images to numpy array : 0.3826780319213867 time to import caffe and check if the image exist : 0.015446901321411133 time to convert the images to numpy array : 0.41606712341308594 time to import caffe and check if the image exist : 0.018240928649902344 time to convert the images to numpy array : 0.4831969738006592 time to import caffe and check if the image exist : 0.015535116195678711 time to convert the images to numpy array : 0.49427223205566406 time to import caffe and check if the image exist : 0.017656326293945312 time to convert the images to numpy array : 0.4929819107055664 time to import caffe and check if the image exist : 0.018590450286865234 time to convert the images to numpy array : 0.5391132831573486 time to import caffe and check if the image exist : 0.011491537094116211 time to convert the images to numpy array : 0.5558943748474121 time to import caffe and check if the image exist : 0.013427019119262695 time to convert the images to numpy array : 0.5550789833068848 time to import caffe and check if the image exist : 0.018246173858642578 time to convert the images to numpy array : 0.553701639175415 total time to convert the images to numpy array : 0.8790020942687988 list photo_ids error: [] list photo_ids correct : [-3655849775, -3655849805, -3653455161, -3655849804, -3653455167, -3653455186, -3655849740, -3655849761, -3655849772, -3655849767, -3655849771, -3655849766, -3653455177, -3653455202, -3653455222, -3655849756, -3655849757, -3655849776, -3655849773, -3655849769, -3655849782, -3655849774, -3655849783, -3655849784, -3655849799, -3653455165, -3653455178, -3653455173, -3653455174, -3653455179, -3653455204, -3655849736, -3655849752, -3655849817, -3653455197, -3653455195, -3655849816, -3653455196, -3653455207, -3653455230, -3653455224, -3655849796, -3655849743, -3655849737, -3655849754, -3655849765, -3655849755, -3655849759, -3655849753, -3655849762, -3655849758, -3655849803, -3655849810, -3653455170, -3653455201, -3655849730, -3655849732, -3655849728, -3655849738, -3655849741, -3653455176, -3653455185, -3653455180, -3653455172, -3653455181, -3653455183, -3653455193, -3653455209, -3653455194, -3655849795, -3655849794, -3655849789, -3655849788, -3655849798, -3655849811, -3653455162, -3653455171, -3655849814, -3655849731, -3655849748, -3655849760, -3655849777, -3655849778, -3655849806, -3655849815, -3655849747, -3655849742] number of photos to traite : 87 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4802 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4583 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.1813323497772217 time used to do the prediction : 0.28086066246032715 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 2481 wait 20 seconds l 3637 free memory gpu now : 2481 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4579 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.0627539157867432 time used to do the prediction : 0.267228364944458 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.5272369384765625e-05 save missing photos in datou_result : time spend for datou_step_exec : 31.005009412765503 time spend to save output : 0.0008838176727294922 total time spend for step 3 : 31.005893230438232 step4:merge_mask_thcl_custom Wed Feb 5 12:02:37 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 164 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1334711703, 1334711699, 1334711697, 1334696715, 1334696655, 1334696538, 1334513930, 1334513927, 1334497467, 1334497463] Looping around the photos to save general results len do output : 10 /1334711703Didn't retrieve data .Didn't retrieve data . /1334711699Didn't retrieve data .Didn't retrieve data . /1334711697Didn't retrieve data .Didn't retrieve data . /1334696715Didn't retrieve data .Didn't retrieve data . /1334696655Didn't retrieve data .Didn't retrieve data . /1334696538Didn't retrieve data .Didn't retrieve data . /1334513930Didn't retrieve data .Didn't retrieve data . /1334513927Didn't retrieve data .Didn't retrieve data . /1334497467Didn't retrieve data .Didn't retrieve data . /1334497463Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711703', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711699', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711697', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696715', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696655', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696538', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513930', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513927', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497467', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497463', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013929367065429688 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11054778099060059 time spend to save output : 0.014462947845458984 total time spend for step 4 : 0.12501072883605957 step5:rle_unique_nms_with_priority Wed Feb 5 12:02:37 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 6 nb_hashtags : 3 time to prepare the origin masks : 6.157535791397095 time for calcul the mask position with numpy : 0.37400078773498535 nb_pixel_total : 3548944 time to create 1 rle with new method : 0.7244722843170166 time for calcul the mask position with numpy : 0.020882606506347656 nb_pixel_total : 31850 time to create 1 rle with old method : 0.03427600860595703 time for calcul the mask position with numpy : 0.021552562713623047 nb_pixel_total : 8119 time to create 1 rle with old method : 0.00901341438293457 time for calcul the mask position with numpy : 0.024086952209472656 nb_pixel_total : 284955 time to create 1 rle with new method : 0.6227965354919434 time for calcul the mask position with numpy : 0.03320574760437012 nb_pixel_total : 2628 time to create 1 rle with old method : 0.0029296875 time for calcul the mask position with numpy : 0.4132509231567383 nb_pixel_total : 3049269 time to create 1 rle with new method : 0.4607698917388916 time for calcul the mask position with numpy : 0.03434324264526367 nb_pixel_total : 124475 time to create 1 rle with old method : 0.13417434692382812 create new chi : 2.993490695953369 time to delete rle : 0.009550333023071289 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 11807 TO DO : save crop sub photo not yet done ! save time : 2.9648473262786865 nb_obj : 16 nb_hashtags : 5 time to prepare the origin masks : 9.35077166557312 time for calcul the mask position with numpy : 0.5338618755340576 nb_pixel_total : 6001038 time to create 1 rle with new method : 0.6786317825317383 time for calcul the mask position with numpy : 0.020920753479003906 nb_pixel_total : 112599 time to create 1 rle with old method : 0.11148834228515625 time for calcul the mask position with numpy : 0.02086043357849121 nb_pixel_total : 64965 time to create 1 rle with old method : 0.06825900077819824 time for calcul the mask position with numpy : 0.02050304412841797 nb_pixel_total : 93867 time to create 1 rle with old method : 0.09701657295227051 time for calcul the mask position with numpy : 0.020491361618041992 nb_pixel_total : 5315 time to create 1 rle with old method : 0.005740165710449219 time for calcul the mask position with numpy : 0.022159814834594727 nb_pixel_total : 240267 time to create 1 rle with new method : 1.075448989868164 time for calcul the mask position with numpy : 0.020826339721679688 nb_pixel_total : 1166 time to create 1 rle with old method : 0.0013432502746582031 time for calcul the mask position with numpy : 0.02132701873779297 nb_pixel_total : 312 time to create 1 rle with old method : 0.00038051605224609375 time for calcul the mask position with numpy : 0.020875215530395508 nb_pixel_total : 134975 time to create 1 rle with old method : 0.13741493225097656 time for calcul the mask position with numpy : 0.02275538444519043 nb_pixel_total : 215766 time to create 1 rle with new method : 0.8290665149688721 time for calcul the mask position with numpy : 0.020672082901000977 nb_pixel_total : 58673 time to create 1 rle with old method : 0.06080484390258789 time for calcul the mask position with numpy : 0.029078006744384766 nb_pixel_total : 6463 time to create 1 rle with old method : 0.006743431091308594 time for calcul the mask position with numpy : 0.033480167388916016 nb_pixel_total : 48144 time to create 1 rle with old method : 0.050681352615356445 time for calcul the mask position with numpy : 0.03278350830078125 nb_pixel_total : 3855 time to create 1 rle with old method : 0.004485368728637695 time for calcul the mask position with numpy : 0.032942771911621094 nb_pixel_total : 5874 time to create 1 rle with old method : 0.0067768096923828125 time for calcul the mask position with numpy : 0.03194928169250488 nb_pixel_total : 2777 time to create 1 rle with old method : 0.0031626224517822266 time for calcul the mask position with numpy : 0.02863311767578125 nb_pixel_total : 54184 time to create 1 rle with old method : 0.05738067626953125 create new chi : 4.210691690444946 time to delete rle : 0.0009732246398925781 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 10857 TO DO : save crop sub photo not yet done ! save time : 2.295226573944092 nb_obj : 15 nb_hashtags : 5 time to prepare the origin masks : 9.256213665008545 time for calcul the mask position with numpy : 0.6673581600189209 nb_pixel_total : 6329026 time to create 1 rle with new method : 0.758467435836792 time for calcul the mask position with numpy : 0.03305244445800781 nb_pixel_total : 85684 time to create 1 rle with old method : 0.0847010612487793 time for calcul the mask position with numpy : 0.03315234184265137 nb_pixel_total : 305 time to create 1 rle with old method : 0.0003864765167236328 time for calcul the mask position with numpy : 0.03292250633239746 nb_pixel_total : 1548 time to create 1 rle with old method : 0.001832723617553711 time for calcul the mask position with numpy : 0.019646167755126953 nb_pixel_total : 2917 time to create 1 rle with old method : 0.0031003952026367188 time for calcul the mask position with numpy : 0.020588159561157227 nb_pixel_total : 62984 time to create 1 rle with old method : 0.06396842002868652 time for calcul the mask position with numpy : 0.02036571502685547 nb_pixel_total : 69274 time to create 1 rle with old method : 0.06951594352722168 time for calcul the mask position with numpy : 0.02032637596130371 nb_pixel_total : 48989 time to create 1 rle with old method : 0.04886293411254883 time for calcul the mask position with numpy : 0.020182132720947266 nb_pixel_total : 82064 time to create 1 rle with old method : 0.08228111267089844 time for calcul the mask position with numpy : 0.020246028900146484 nb_pixel_total : 44245 time to create 1 rle with old method : 0.046033620834350586 time for calcul the mask position with numpy : 0.020323514938354492 nb_pixel_total : 15077 time to create 1 rle with old method : 0.01623392105102539 time for calcul the mask position with numpy : 0.020119190216064453 nb_pixel_total : 25336 time to create 1 rle with old method : 0.025799036026000977 time for calcul the mask position with numpy : 0.020351171493530273 nb_pixel_total : 20146 time to create 1 rle with old method : 0.020655155181884766 time for calcul the mask position with numpy : 0.020819902420043945 nb_pixel_total : 39309 time to create 1 rle with old method : 0.04233980178833008 time for calcul the mask position with numpy : 0.024897336959838867 nb_pixel_total : 93841 time to create 1 rle with old method : 0.10852360725402832 time for calcul the mask position with numpy : 0.02674126625061035 nb_pixel_total : 129495 time to create 1 rle with old method : 0.1380767822265625 create new chi : 2.572129011154175 time to delete rle : 0.000988006591796875 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 12079 TO DO : save crop sub photo not yet done ! save time : 6.492004871368408 nb_obj : 21 nb_hashtags : 8 time to prepare the origin masks : 13.481156349182129 time for calcul the mask position with numpy : 0.2786128520965576 nb_pixel_total : 5885700 time to create 1 rle with new method : 0.3566749095916748 time for calcul the mask position with numpy : 0.0281221866607666 nb_pixel_total : 42129 time to create 1 rle with old method : 0.04350590705871582 time for calcul the mask position with numpy : 0.03723621368408203 nb_pixel_total : 78143 time to create 1 rle with old method : 0.08235430717468262 time for calcul the mask position with numpy : 0.032948970794677734 nb_pixel_total : 122619 time to create 1 rle with old method : 0.12375402450561523 time for calcul the mask position with numpy : 0.03446602821350098 nb_pixel_total : 44591 time to create 1 rle with old method : 0.04662442207336426 time for calcul the mask position with numpy : 0.03222823143005371 nb_pixel_total : 10121 time to create 1 rle with old method : 0.010395050048828125 time for calcul the mask position with numpy : 0.03224372863769531 nb_pixel_total : 35048 time to create 1 rle with old method : 0.03473091125488281 time for calcul the mask position with numpy : 0.03657341003417969 nb_pixel_total : 355630 time to create 1 rle with new method : 0.848515510559082 time for calcul the mask position with numpy : 0.03413105010986328 nb_pixel_total : 2630 time to create 1 rle with old method : 0.0030100345611572266 time for calcul the mask position with numpy : 0.03448081016540527 nb_pixel_total : 3106 time to create 1 rle with old method : 0.003562450408935547 time for calcul the mask position with numpy : 0.036617279052734375 nb_pixel_total : 147166 time to create 1 rle with old method : 0.15797853469848633 time for calcul the mask position with numpy : 0.0339970588684082 nb_pixel_total : 6182 time to create 1 rle with old method : 0.0068318843841552734 time for calcul the mask position with numpy : 0.03162097930908203 nb_pixel_total : 6514 time to create 1 rle with old method : 0.007184743881225586 time for calcul the mask position with numpy : 0.02965521812438965 nb_pixel_total : 2205 time to create 1 rle with old method : 0.002431154251098633 time for calcul the mask position with numpy : 0.03372693061828613 nb_pixel_total : 2378 time to create 1 rle with old method : 0.0027685165405273438 time for calcul the mask position with numpy : 0.02505660057067871 nb_pixel_total : 23349 time to create 1 rle with old method : 0.02540302276611328 time for calcul the mask position with numpy : 0.03535199165344238 nb_pixel_total : 45821 time to create 1 rle with old method : 0.04850292205810547 time for calcul the mask position with numpy : 0.03504514694213867 nb_pixel_total : 82903 time to create 1 rle with old method : 0.08596301078796387 time for calcul the mask position with numpy : 0.03343963623046875 nb_pixel_total : 34180 time to create 1 rle with old method : 0.03557324409484863 time for calcul the mask position with numpy : 0.03300762176513672 nb_pixel_total : 2117 time to create 1 rle with old method : 0.00229644775390625 time for calcul the mask position with numpy : 0.02590346336364746 nb_pixel_total : 11315 time to create 1 rle with old method : 0.011699676513671875 time for calcul the mask position with numpy : 0.02698826789855957 nb_pixel_total : 106393 time to create 1 rle with old method : 0.13489246368408203 create new chi : 3.101635217666626 time to delete rle : 0.0019371509552001953 batch 1 Loaded 22 chid ids of type : 3726 Number RLEs to save : 14020 TO DO : save crop sub photo not yet done ! save time : 0.7816839218139648 nb_obj : 7 nb_hashtags : 4 time to prepare the origin masks : 5.203459739685059 time for calcul the mask position with numpy : 0.5015308856964111 nb_pixel_total : 3457244 time to create 1 rle with new method : 0.7590618133544922 time for calcul the mask position with numpy : 0.03468036651611328 nb_pixel_total : 57260 time to create 1 rle with old method : 0.06447410583496094 time for calcul the mask position with numpy : 0.03888988494873047 nb_pixel_total : 233051 time to create 1 rle with new method : 0.6923949718475342 time for calcul the mask position with numpy : 0.03770923614501953 nb_pixel_total : 368898 time to create 1 rle with new method : 0.5099761486053467 time for calcul the mask position with numpy : 0.028882741928100586 nb_pixel_total : 64221 time to create 1 rle with old method : 0.07022261619567871 time for calcul the mask position with numpy : 0.025150537490844727 nb_pixel_total : 2766 time to create 1 rle with old method : 0.0030562877655029297 time for calcul the mask position with numpy : 0.15125536918640137 nb_pixel_total : 2751733 time to create 1 rle with new method : 0.9076673984527588 time for calcul the mask position with numpy : 0.03465461730957031 nb_pixel_total : 115067 time to create 1 rle with old method : 0.12178826332092285 create new chi : 4.0946924686431885 time to delete rle : 0.0012559890747070312 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 14631 TO DO : save crop sub photo not yet done ! save time : 1.321714162826538 nb_obj : 14 nb_hashtags : 6 time to prepare the origin masks : 9.10833477973938 time for calcul the mask position with numpy : 0.9134111404418945 nb_pixel_total : 4543433 time to create 1 rle with new method : 0.9207642078399658 time for calcul the mask position with numpy : 0.021534442901611328 nb_pixel_total : 15599 time to create 1 rle with old method : 0.017147302627563477 time for calcul the mask position with numpy : 0.021506309509277344 nb_pixel_total : 19713 time to create 1 rle with old method : 0.021035432815551758 time for calcul the mask position with numpy : 0.021570682525634766 nb_pixel_total : 23396 time to create 1 rle with old method : 0.02532219886779785 time for calcul the mask position with numpy : 0.024944782257080078 nb_pixel_total : 355334 time to create 1 rle with new method : 1.0596535205841064 time for calcul the mask position with numpy : 0.023221969604492188 nb_pixel_total : 69978 time to create 1 rle with old method : 0.07533502578735352 time for calcul the mask position with numpy : 0.02209925651550293 nb_pixel_total : 12059 time to create 1 rle with old method : 0.0131072998046875 time for calcul the mask position with numpy : 0.022167444229125977 nb_pixel_total : 10041 time to create 1 rle with old method : 0.010648488998413086 time for calcul the mask position with numpy : 0.03291893005371094 nb_pixel_total : 1454290 time to create 1 rle with new method : 1.1995117664337158 time for calcul the mask position with numpy : 0.02544093132019043 nb_pixel_total : 390051 time to create 1 rle with new method : 0.33423352241516113 time for calcul the mask position with numpy : 0.021930694580078125 nb_pixel_total : 27372 time to create 1 rle with old method : 0.029524564743041992 time for calcul the mask position with numpy : 0.022966861724853516 nb_pixel_total : 90062 time to create 1 rle with old method : 0.09622859954833984 time for calcul the mask position with numpy : 0.021815776824951172 nb_pixel_total : 4416 time to create 1 rle with old method : 0.0049092769622802734 time for calcul the mask position with numpy : 0.021615982055664062 nb_pixel_total : 31673 time to create 1 rle with old method : 0.033174991607666016 time for calcul the mask position with numpy : 0.02109527587890625 nb_pixel_total : 2823 time to create 1 rle with old method : 0.0031311511993408203 create new chi : 5.190181732177734 time to delete rle : 0.0012884140014648438 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 14372 TO DO : save crop sub photo not yet done ! save time : 0.7863492965698242 nb_obj : 15 nb_hashtags : 5 time to prepare the origin masks : 9.333688259124756 time for calcul the mask position with numpy : 0.24015307426452637 nb_pixel_total : 4973805 time to create 1 rle with new method : 0.7531321048736572 time for calcul the mask position with numpy : 0.037216901779174805 nb_pixel_total : 348078 time to create 1 rle with new method : 0.7744355201721191 time for calcul the mask position with numpy : 0.037682294845581055 nb_pixel_total : 218946 time to create 1 rle with new method : 0.37879014015197754 time for calcul the mask position with numpy : 0.0334012508392334 nb_pixel_total : 295542 time to create 1 rle with new method : 0.49226808547973633 time for calcul the mask position with numpy : 0.029226303100585938 nb_pixel_total : 180 time to create 1 rle with old method : 0.0003452301025390625 time for calcul the mask position with numpy : 0.032210350036621094 nb_pixel_total : 5228 time to create 1 rle with old method : 0.007339954376220703 time for calcul the mask position with numpy : 0.027054548263549805 nb_pixel_total : 1032 time to create 1 rle with old method : 0.0019278526306152344 time for calcul the mask position with numpy : 0.03906893730163574 nb_pixel_total : 18346 time to create 1 rle with old method : 0.023105621337890625 time for calcul the mask position with numpy : 0.033196210861206055 nb_pixel_total : 150353 time to create 1 rle with new method : 0.697664737701416 time for calcul the mask position with numpy : 0.037732601165771484 nb_pixel_total : 22578 time to create 1 rle with old method : 0.024763822555541992 time for calcul the mask position with numpy : 0.0430455207824707 nb_pixel_total : 875041 time to create 1 rle with new method : 0.3858528137207031 time for calcul the mask position with numpy : 0.03397631645202637 nb_pixel_total : 22 time to create 1 rle with old method : 7.486343383789062e-05 time for calcul the mask position with numpy : 0.03259110450744629 nb_pixel_total : 1557 time to create 1 rle with old method : 0.0017216205596923828 time for calcul the mask position with numpy : 0.031638383865356445 nb_pixel_total : 46926 time to create 1 rle with old method : 0.04836392402648926 time for calcul the mask position with numpy : 0.029359817504882812 nb_pixel_total : 15828 time to create 1 rle with old method : 0.017072200775146484 time for calcul the mask position with numpy : 0.02875351905822754 nb_pixel_total : 76778 time to create 1 rle with old method : 0.10606622695922852 create new chi : 4.613565444946289 time to delete rle : 0.0018267631530761719 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 13464 TO DO : save crop sub photo not yet done ! save time : 0.6983113288879395 nb_obj : 14 nb_hashtags : 7 time to prepare the origin masks : 10.918707847595215 time for calcul the mask position with numpy : 1.0361642837524414 nb_pixel_total : 6243884 time to create 1 rle with new method : 1.7266845703125 time for calcul the mask position with numpy : 0.021781444549560547 nb_pixel_total : 33654 time to create 1 rle with old method : 0.0357823371887207 time for calcul the mask position with numpy : 0.02402186393737793 nb_pixel_total : 95946 time to create 1 rle with old method : 0.11007881164550781 time for calcul the mask position with numpy : 0.021086931228637695 nb_pixel_total : 13846 time to create 1 rle with old method : 0.018295764923095703 time for calcul the mask position with numpy : 0.025246858596801758 nb_pixel_total : 20523 time to create 1 rle with old method : 0.024814367294311523 time for calcul the mask position with numpy : 0.029876708984375 nb_pixel_total : 36225 time to create 1 rle with old method : 0.04292750358581543 time for calcul the mask position with numpy : 0.022428035736083984 nb_pixel_total : 36985 time to create 1 rle with old method : 0.04247760772705078 time for calcul the mask position with numpy : 0.02162647247314453 nb_pixel_total : 30125 time to create 1 rle with old method : 0.03173089027404785 time for calcul the mask position with numpy : 0.021526098251342773 nb_pixel_total : 13566 time to create 1 rle with old method : 0.014270782470703125 time for calcul the mask position with numpy : 0.021292924880981445 nb_pixel_total : 23817 time to create 1 rle with old method : 0.029216289520263672 time for calcul the mask position with numpy : 0.022404193878173828 nb_pixel_total : 19689 time to create 1 rle with old method : 0.03175687789916992 time for calcul the mask position with numpy : 0.026874542236328125 nb_pixel_total : 256649 time to create 1 rle with new method : 0.6969552040100098 time for calcul the mask position with numpy : 0.03198599815368652 nb_pixel_total : 15995 time to create 1 rle with old method : 0.017040252685546875 time for calcul the mask position with numpy : 0.033174991607666016 nb_pixel_total : 35842 time to create 1 rle with old method : 0.0376737117767334 time for calcul the mask position with numpy : 0.03422188758850098 nb_pixel_total : 173494 time to create 1 rle with new method : 0.7414572238922119 create new chi : 5.081528425216675 time to delete rle : 0.0010998249053955078 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 11164 TO DO : save crop sub photo not yet done ! save time : 0.6232316493988037 nb_obj : 24 nb_hashtags : 4 time to prepare the origin masks : 12.804745197296143 time for calcul the mask position with numpy : 0.3998115062713623 nb_pixel_total : 5532145 time to create 1 rle with new method : 0.6689159870147705 time for calcul the mask position with numpy : 0.033541202545166016 nb_pixel_total : 4413 time to create 1 rle with old method : 0.0045261383056640625 time for calcul the mask position with numpy : 0.03489422798156738 nb_pixel_total : 105756 time to create 1 rle with old method : 0.10796213150024414 time for calcul the mask position with numpy : 0.03625321388244629 nb_pixel_total : 372428 time to create 1 rle with new method : 0.6665220260620117 time for calcul the mask position with numpy : 0.03242754936218262 nb_pixel_total : 3746 time to create 1 rle with old method : 0.0040700435638427734 time for calcul the mask position with numpy : 0.03160595893859863 nb_pixel_total : 109686 time to create 1 rle with old method : 0.11667513847351074 time for calcul the mask position with numpy : 0.03109145164489746 nb_pixel_total : 33465 time to create 1 rle with old method : 0.035501718521118164 time for calcul the mask position with numpy : 0.030527830123901367 nb_pixel_total : 55062 time to create 1 rle with old method : 0.05804252624511719 time for calcul the mask position with numpy : 0.031232833862304688 nb_pixel_total : 17824 time to create 1 rle with old method : 0.01860976219177246 time for calcul the mask position with numpy : 0.02880120277404785 nb_pixel_total : 94653 time to create 1 rle with old method : 0.09999847412109375 time for calcul the mask position with numpy : 0.02859044075012207 nb_pixel_total : 2159 time to create 1 rle with old method : 0.0022716522216796875 time for calcul the mask position with numpy : 0.030463695526123047 nb_pixel_total : 56209 time to create 1 rle with old method : 0.05973505973815918 time for calcul the mask position with numpy : 0.034635305404663086 nb_pixel_total : 5307 time to create 1 rle with old method : 0.006071567535400391 time for calcul the mask position with numpy : 0.03448367118835449 nb_pixel_total : 3036 time to create 1 rle with old method : 0.0034046173095703125 time for calcul the mask position with numpy : 0.03513455390930176 nb_pixel_total : 733 time to create 1 rle with old method : 0.0008633136749267578 time for calcul the mask position with numpy : 0.03496050834655762 nb_pixel_total : 10444 time to create 1 rle with old method : 0.011904716491699219 time for calcul the mask position with numpy : 0.03770089149475098 nb_pixel_total : 7818 time to create 1 rle with old method : 0.008250236511230469 time for calcul the mask position with numpy : 0.033453941345214844 nb_pixel_total : 11616 time to create 1 rle with old method : 0.012171506881713867 time for calcul the mask position with numpy : 0.03451800346374512 nb_pixel_total : 159612 time to create 1 rle with new method : 0.4451465606689453 time for calcul the mask position with numpy : 0.03263282775878906 nb_pixel_total : 25350 time to create 1 rle with old method : 0.026467561721801758 time for calcul the mask position with numpy : 0.026724576950073242 nb_pixel_total : 40286 time to create 1 rle with old method : 0.04220247268676758 time for calcul the mask position with numpy : 0.026170015335083008 nb_pixel_total : 7799 time to create 1 rle with old method : 0.008285284042358398 time for calcul the mask position with numpy : 0.03399229049682617 nb_pixel_total : 338915 time to create 1 rle with new method : 0.5482223033905029 time for calcul the mask position with numpy : 0.03350496292114258 nb_pixel_total : 1711 time to create 1 rle with old method : 0.10127806663513184 time for calcul the mask position with numpy : 0.038205623626708984 nb_pixel_total : 50067 time to create 1 rle with old method : 0.0612483024597168 create new chi : 4.415949583053589 time to delete rle : 0.001333475112915039 batch 1 Loaded 29 chid ids of type : 3726 Number RLEs to save : 15005 TO DO : save crop sub photo not yet done ! save time : 1.0165283679962158 nb_obj : 17 nb_hashtags : 2 time to prepare the origin masks : 9.051860570907593 time for calcul the mask position with numpy : 0.7874784469604492 nb_pixel_total : 6875892 time to create 1 rle with new method : 0.30377697944641113 time for calcul the mask position with numpy : 0.024446964263916016 nb_pixel_total : 61337 time to create 1 rle with old method : 0.06648421287536621 time for calcul the mask position with numpy : 0.023510217666625977 nb_pixel_total : 24346 time to create 1 rle with old method : 0.024407148361206055 time for calcul the mask position with numpy : 0.02421855926513672 nb_pixel_total : 1362 time to create 1 rle with old method : 0.0015304088592529297 time for calcul the mask position with numpy : 0.023865461349487305 nb_pixel_total : 4432 time to create 1 rle with old method : 0.004587650299072266 time for calcul the mask position with numpy : 0.024587392807006836 nb_pixel_total : 1008 time to create 1 rle with old method : 0.001104593276977539 time for calcul the mask position with numpy : 0.025069475173950195 nb_pixel_total : 4109 time to create 1 rle with old method : 0.004122495651245117 time for calcul the mask position with numpy : 0.02434563636779785 nb_pixel_total : 2872 time to create 1 rle with old method : 0.0031366348266601562 time for calcul the mask position with numpy : 0.0377802848815918 nb_pixel_total : 1612 time to create 1 rle with old method : 0.0017635822296142578 time for calcul the mask position with numpy : 0.04118919372558594 nb_pixel_total : 5545 time to create 1 rle with old method : 0.00615692138671875 time for calcul the mask position with numpy : 0.04088425636291504 nb_pixel_total : 3892 time to create 1 rle with old method : 0.004335165023803711 time for calcul the mask position with numpy : 0.03996086120605469 nb_pixel_total : 25859 time to create 1 rle with old method : 0.026796579360961914 time for calcul the mask position with numpy : 0.03789234161376953 nb_pixel_total : 3873 time to create 1 rle with old method : 0.00407862663269043 time for calcul the mask position with numpy : 0.03725481033325195 nb_pixel_total : 3399 time to create 1 rle with old method : 0.0035965442657470703 time for calcul the mask position with numpy : 0.041330575942993164 nb_pixel_total : 9484 time to create 1 rle with old method : 0.009539365768432617 time for calcul the mask position with numpy : 0.03713560104370117 nb_pixel_total : 8833 time to create 1 rle with old method : 0.009333133697509766 time for calcul the mask position with numpy : 0.03690695762634277 nb_pixel_total : 6166 time to create 1 rle with old method : 0.007598876953125 time for calcul the mask position with numpy : 0.04013681411743164 nb_pixel_total : 6219 time to create 1 rle with old method : 0.0070018768310546875 create new chi : 1.88374662399292 time to delete rle : 0.0011129379272460938 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 6622 TO DO : save crop sub photo not yet done ! save time : 0.3968234062194824 map_output_result : {1334711703: (0.0, 'Should be the crop_list due to order', 0.0), 1334711699: (0.0, 'Should be the crop_list due to order', 0.0), 1334711697: (0.0, 'Should be the crop_list due to order', 0.0), 1334696715: (0.0, 'Should be the crop_list due to order', 0.0), 1334696655: (0.0, 'Should be the crop_list due to order', 0.0), 1334696538: (0.0, 'Should be the crop_list due to order', 0.0), 1334513930: (0.0, 'Should be the crop_list due to order', 0.0), 1334513927: (0.0, 'Should be the crop_list due to order', 0.0), 1334497467: (0.0, 'Should be the crop_list due to order', 0.0), 1334497463: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1334711703, 1334711699, 1334711697, 1334696715, 1334696655, 1334696538, 1334513930, 1334513927, 1334497467, 1334497463] Looping around the photos to save general results len do output : 10 /1334711703.Didn't retrieve data . /1334711699.Didn't retrieve data . /1334711697.Didn't retrieve data . /1334696715.Didn't retrieve data . /1334696655.Didn't retrieve data . /1334696538.Didn't retrieve data . /1334513930.Didn't retrieve data . /1334513927.Didn't retrieve data . /1334497467.Didn't retrieve data . /1334497463.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711703', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711699', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711697', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696715', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696655', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696538', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513930', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513927', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497467', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497463', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014475345611572266 save_final save missing photos in datou_result : time spend for datou_step_exec : 151.13938665390015 time spend to save output : 0.014915704727172852 total time spend for step 5 : 151.15430235862732 step6:crop_condition Wed Feb 5 12:05:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 169 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738753512_3550052 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.7127306461334229 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 12 About to insert : list_path_to_insert length 12 new photo from crops ! About to upload 12 photos upload in portfolio : 4869462 init cache_photo without model_param we have 12 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738753520_3550052 we have uploaded 12 photos in the portfolio 4869462 time of upload the photos Elapsed time : 3.269756317138672 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738753525_3550052 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.0113317966461182 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738753528_3550052 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8873393535614014 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738753533_3550052 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.9032435417175293 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738753538_3550052 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7899291515350342 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738753540_3550052 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8123176097869873 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1334711703, 1334711699, 1334711697, 1334696715, 1334696655, 1334696538, 1334513930, 1334513927, 1334497467, 1334497463] Looping around the photos to save general results len do output : 33 /1334905695Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905696Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905697Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905703Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905704Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905705Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905706Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905707Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905708Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905709Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905710Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905711Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905712Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905713Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905714Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905715Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905716Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905717Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905718Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905727Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905728Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905737Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905738Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905739Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905740Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905741Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905742Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905743Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905744Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905747Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905748Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905752Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334905753Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711703', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711699', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711697', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696715', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696655', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696538', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513930', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513927', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497467', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497463', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 109 time used for this insertion : 0.03292965888977051 save_final save missing photos in datou_result : time spend for datou_step_exec : 34.060717821121216 time spend to save output : 0.03439927101135254 total time spend for step 6 : 34.09511709213257 step7:ventilate_hashtags_in_portfolio Wed Feb 5 12:05:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20254669 get user id for portfolio 20254669 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254669 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','metal','cartonnette','mal_croppe','Carton_gris','flou','Carton_brun','kraft','plastique','Teint_Dans_La_Masse','autre_refus','papier')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254669 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','metal','cartonnette','mal_croppe','Carton_gris','flou','Carton_brun','kraft','plastique','Teint_Dans_La_Masse','autre_refus','papier')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254669 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','metal','cartonnette','mal_croppe','Carton_gris','flou','Carton_brun','kraft','plastique','Teint_Dans_La_Masse','autre_refus','papier')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20255897,20255898,20255899,20255900,20255901,20255902,20255903,20255904,20255905,20255906,20255907,20255908?tags=environnement,metal,cartonnette,mal_croppe,Carton_gris,flou,Carton_brun,kraft,plastique,Teint_Dans_La_Masse,autre_refus,papier Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1334711703, 1334711699, 1334711697, 1334696715, 1334696655, 1334696538, 1334513930, 1334513927, 1334497467, 1334497463] Looping around the photos to save general results len do output : 1 /20254669. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711703', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711699', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711697', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696715', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696655', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696538', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513930', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513927', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497467', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497463', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01384115219116211 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.8127355575561523 time spend to save output : 0.014124870300292969 total time spend for step 7 : 0.8268604278564453 step8:final Wed Feb 5 12:05:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1334711703: ('0.07156960103454109',), 1334711699: ('0.07156960103454109',), 1334711697: ('0.07156960103454109',), 1334696715: ('0.07156960103454109',), 1334696655: ('0.07156960103454109',), 1334696538: ('0.07156960103454109',), 1334513930: ('0.07156960103454109',), 1334513927: ('0.07156960103454109',), 1334497467: ('0.07156960103454109',), 1334497463: ('0.07156960103454109',)} new output for save of step final : {1334711703: ('0.07156960103454109',), 1334711699: ('0.07156960103454109',), 1334711697: ('0.07156960103454109',), 1334696715: ('0.07156960103454109',), 1334696655: ('0.07156960103454109',), 1334696538: ('0.07156960103454109',), 1334513930: ('0.07156960103454109',), 1334513927: ('0.07156960103454109',), 1334497467: ('0.07156960103454109',), 1334497463: ('0.07156960103454109',)} [1334711703, 1334711699, 1334711697, 1334696715, 1334696655, 1334696538, 1334513930, 1334513927, 1334497467, 1334497463] Looping around the photos to save general results len do output : 10 /1334711703.Didn't retrieve data . /1334711699.Didn't retrieve data . /1334711697.Didn't retrieve data . /1334696715.Didn't retrieve data . /1334696655.Didn't retrieve data . /1334696538.Didn't retrieve data . /1334513930.Didn't retrieve data . /1334513927.Didn't retrieve data . /1334497467.Didn't retrieve data . /1334497463.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711703', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711699', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711697', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696715', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696655', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696538', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513930', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513927', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497467', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497463', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013458251953125 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.09991049766540527 time spend to save output : 0.013979196548461914 total time spend for step 8 : 0.11388969421386719 step9:velours_tree Wed Feb 5 12:05:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.05801653861999512 time spend to save output : 3.814697265625e-05 total time spend for step 9 : 0.05805468559265137 step10:send_mail_cod Wed Feb 5 12:05:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20254669_05-02-2025_12_05_43.pdf 20255898 change filename to text .imagette202558981738753543 20255899 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202558991738753544 20255900 imagette202559001738753546 20255902 imagette202559021738753546 20255904 change filename to text .change filename to text .change filename to text .imagette202559041738753546 20255905 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202559051738753546 20255907 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202559071738753547 20255901 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202559011738753549 20255903 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202559031738753550 20255906 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202559061738753551 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20254669 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20255897,20255898,20255899,20255900,20255901,20255902,20255903,20255904,20255905,20255906,20255907,20255908?tags=environnement,metal,cartonnette,mal_croppe,Carton_gris,flou,Carton_brun,kraft,plastique,Teint_Dans_La_Masse,autre_refus,papier your option no_mail is active, we will not send the real mail to your client args[1334711703] : ('0.07156960103454109',) no score found for photo 1334711703 We are sending mail with results at report@fotonower.com args[1334711699] : ('0.07156960103454109',) no score found for photo 1334711699 We are sending mail with results at report@fotonower.com args[1334711697] : ('0.07156960103454109',) no score found for photo 1334711697 We are sending mail with results at report@fotonower.com args[1334696715] : ('0.07156960103454109',) no score found for photo 1334696715 We are sending mail with results at report@fotonower.com args[1334696655] : ('0.07156960103454109',) no score found for photo 1334696655 We are sending mail with results at report@fotonower.com args[1334696538] : ('0.07156960103454109',) no score found for photo 1334696538 We are sending mail with results at report@fotonower.com args[1334513930] : ('0.07156960103454109',) no score found for photo 1334513930 We are sending mail with results at report@fotonower.com args[1334513927] : ('0.07156960103454109',) no score found for photo 1334513927 We are sending mail with results at report@fotonower.com args[1334497467] : ('0.07156960103454109',) no score found for photo 1334497467 We are sending mail with results at report@fotonower.com args[1334497463] : ('0.07156960103454109',) no score found for photo 1334497463 We are sending mail with results at report@fotonower.com refus_total : 0.07156960103454109 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20254669 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1334496996,1334696715,1334497467,1334497463,1334497458,1334497454,1334497450,1334711697,1334497444,1334497441,1334497438,1334711699,1334497411,1334497400,1334497358,1334497355,1334497331,1334497302,1334496991,1334497008) Found this number of photos: 20 begin to download photo : 1334496996 begin to download photo : 1334497454 begin to download photo : 1334497438 begin to download photo : 1334497355 download finish for photo 1334497454 begin to download photo : 1334497450 download finish for photo 1334497438 begin to download photo : 1334711699 download finish for photo 1334496996 begin to download photo : 1334696715 download finish for photo 1334497355 begin to download photo : 1334497331 download finish for photo 1334497450 begin to download photo : 1334711697 download finish for photo 1334711699 begin to download photo : 1334497411 download finish for photo 1334696715 begin to download photo : 1334497467 download finish for photo 1334497331 begin to download photo : 1334497302 download finish for photo 1334711697 begin to download photo : 1334497444 download finish for photo 1334497411 begin to download photo : 1334497400 download finish for photo 1334497467 begin to download photo : 1334497463 download finish for photo 1334497400 begin to download photo : 1334497358 download finish for photo 1334497302 begin to download photo : 1334496991 download finish for photo 1334497444 begin to download photo : 1334497441 download finish for photo 1334497463 begin to download photo : 1334497458 download finish for photo 1334497358 download finish for photo 1334497458 download finish for photo 1334496991 begin to download photo : 1334497008 download finish for photo 1334497441 download finish for photo 1334497008 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254669_05-02-2025_12_05_43.pdf results_Auto_P20254669_05-02-2025_12_05_43.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254669_05-02-2025_12_05_43.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20254669','results_Auto_P20254669_05-02-2025_12_05_43.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254669_05-02-2025_12_05_43.pdf','pdf','','1.33','0.07156960103454109') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1334711703, 1334711699, 1334711697, 1334696715, 1334696655, 1334696538, 1334513930, 1334513927, 1334497467, 1334497463] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711703', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711699', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711697', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696715', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696655', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696538', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513930', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513927', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497467', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497463', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.017160654067993164 save_final save missing photos in datou_result : time spend for datou_step_exec : 13.948930978775024 time spend to save output : 0.01760387420654297 total time spend for step 10 : 13.966534852981567 step11:split_time_score Wed Feb 5 12:05:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('09', 23), ('10', 5)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 04022025 20254669 Nombre de photos uploadées : 28 / 23040 (0%) 04022025 20254669 Nombre de photos taguées (types de déchets): 0 / 28 (0%) 04022025 20254669 Nombre de photos taguées (volume) : 0 / 28 (0%) elapsed_time : load_data_split_time_score 2.1457672119140625e-06 elapsed_time : order_list_meta_photo_and_scores 6.4373016357421875e-06 ???????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0015478134155273438 elapsed_time : insert_dashboard_record_day_entry 0.03273296356201172 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254648 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254650 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254653 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254655 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254657 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254660 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254661 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254663 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254664 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254666 order by id desc limit 1 Qualite : 0.03827887555059262 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20208806_04-02-2025_12_29_58.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20208806 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20208806 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254668 order by id desc limit 1 Qualite : 0.07156960103454109 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254669_05-02-2025_12_05_43.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254669 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254669 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254671 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254672 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254674 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254675 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254688 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254690 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254691 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254692 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254693 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254694 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254695 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254696 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254697 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254698 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254699 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254700 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254701 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254716 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254727 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254728 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254729 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254730 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254731 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254732 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254733 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254734 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254735 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254736 order by id desc limit 1 Qualite : 0.08109735543377233 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254737_05-02-2025_11_58_05.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254737 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254737 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254738 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254739 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254740 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254741 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254742 order by id desc limit 1 Qualite : 0.09993071763572346 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254767_05-02-2025_11_50_14.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254767 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254767 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254768 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254769 order by id desc limit 1 Qualite : 0.2029836809337933 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254770_05-02-2025_11_49_33.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254770 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254770 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254771 order by id desc limit 1 Qualite : 0.07408981285512192 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254772_05-02-2025_11_59_52.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254772 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254772 AND mptpi.`type`=4207 To do Qualite : 0.09660837363298684 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254773_05-02-2025_11_59_27.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254773 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254773 AND mptpi.`type`=4207 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'04022025': {'nb_upload': 28, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1334711703, 1334711699, 1334711697, 1334696715, 1334696655, 1334696538, 1334513930, 1334513927, 1334497467, 1334497463] Looping around the photos to save general results len do output : 1 /20254669Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711703', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711699', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334711697', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696715', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696655', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334696538', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513930', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334513927', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497467', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', '20254669', '1334497463', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013507366180419922 save_final save missing photos in datou_result : time spend for datou_step_exec : 36.315006494522095 time spend to save output : 0.013725757598876953 total time spend for step 11 : 36.32873225212097 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 156.38user 126.31system 6:10.23elapsed 76%CPU (0avgtext+0avgdata 4440108maxresident)k 511904inputs+163608outputs (6558major+12410767minor)pagefaults 0swaps