python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3524430 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 39>10 please execute split_portfolio.py -i 20254737 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2551122'] with mtr_portfolio_ids : ['20254737'] and first list_photo_ids : [] new path : /proc/3524430/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.436762809753418 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Wed Feb 5 11:54:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-05 11:54:33.671620: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-05 11:54:33.699105: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-05 11:54:33.700663: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f6834000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-05 11:54:33.700699: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-05 11:54:33.702991: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-05 11:54:33.933235: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1002abe0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-05 11:54:33.933275: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-05 11:54:33.934568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 11:54:33.934913: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 11:54:33.937537: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 11:54:33.940027: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 11:54:33.940456: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 11:54:33.943501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 11:54:33.944483: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 11:54:33.948274: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 11:54:33.949729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 11:54:33.949784: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 11:54:33.950555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-05 11:54:33.950570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-05 11:54:33.950579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-05 11:54:33.952032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-05 11:54:34.208872: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 11:54:34.208963: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 11:54:34.208984: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 11:54:34.209003: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 11:54:34.209021: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 11:54:34.209038: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 11:54:34.209055: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 11:54:34.209073: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 11:54:34.210656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 11:54:34.211921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 11:54:34.211956: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 11:54:34.211975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 11:54:34.211992: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 11:54:34.212009: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 11:54:34.212026: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 11:54:34.212042: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 11:54:34.212059: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 11:54:34.213607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 11:54:34.213637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-05 11:54:34.213647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-05 11:54:34.213657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-05 11:54:34.215292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-05 11:54:41.609173: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 11:54:41.762306: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 25 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 24 Detection mask done ! Trying to reset tf kernel 3524770 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5485 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10774 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.001117706298828125 nb_pixel_total : 47829 time to create 1 rle with old method : 0.05285286903381348 length of segment : 203 time for calcul the mask position with numpy : 0.0004818439483642578 nb_pixel_total : 29845 time to create 1 rle with old method : 0.033111572265625 length of segment : 350 time for calcul the mask position with numpy : 0.00015234947204589844 nb_pixel_total : 8023 time to create 1 rle with old method : 0.008793830871582031 length of segment : 81 time for calcul the mask position with numpy : 0.0017757415771484375 nb_pixel_total : 83212 time to create 1 rle with old method : 0.08366966247558594 length of segment : 302 time for calcul the mask position with numpy : 0.00021386146545410156 nb_pixel_total : 10098 time to create 1 rle with old method : 0.010775327682495117 length of segment : 105 time for calcul the mask position with numpy : 0.0008332729339599609 nb_pixel_total : 62633 time to create 1 rle with old method : 0.06358027458190918 length of segment : 406 time for calcul the mask position with numpy : 0.00027489662170410156 nb_pixel_total : 12944 time to create 1 rle with old method : 0.013678789138793945 length of segment : 215 time for calcul the mask position with numpy : 0.0001270771026611328 nb_pixel_total : 6024 time to create 1 rle with old method : 0.006586551666259766 length of segment : 111 time for calcul the mask position with numpy : 0.0007522106170654297 nb_pixel_total : 52792 time to create 1 rle with old method : 0.05498385429382324 length of segment : 292 time for calcul the mask position with numpy : 0.0006055831909179688 nb_pixel_total : 32240 time to create 1 rle with old method : 0.03343462944030762 length of segment : 263 time for calcul the mask position with numpy : 0.0007879734039306641 nb_pixel_total : 38289 time to create 1 rle with old method : 0.0402674674987793 length of segment : 346 time for calcul the mask position with numpy : 0.0001678466796875 nb_pixel_total : 3153 time to create 1 rle with old method : 0.003656625747680664 length of segment : 85 time for calcul the mask position with numpy : 0.001894235610961914 nb_pixel_total : 84779 time to create 1 rle with old method : 0.08667850494384766 length of segment : 596 time for calcul the mask position with numpy : 0.0011320114135742188 nb_pixel_total : 75742 time to create 1 rle with old method : 0.07448697090148926 length of segment : 373 time for calcul the mask position with numpy : 0.0015223026275634766 nb_pixel_total : 72904 time to create 1 rle with old method : 0.08479905128479004 length of segment : 383 time for calcul the mask position with numpy : 0.0006344318389892578 nb_pixel_total : 32633 time to create 1 rle with old method : 0.03474140167236328 length of segment : 271 time for calcul the mask position with numpy : 0.006352663040161133 nb_pixel_total : 387379 time to create 1 rle with new method : 0.017673730850219727 length of segment : 499 time for calcul the mask position with numpy : 0.0007798671722412109 nb_pixel_total : 26404 time to create 1 rle with old method : 0.029021024703979492 length of segment : 196 time for calcul the mask position with numpy : 0.0005860328674316406 nb_pixel_total : 28656 time to create 1 rle with old method : 0.030141353607177734 length of segment : 256 time for calcul the mask position with numpy : 0.0007305145263671875 nb_pixel_total : 60885 time to create 1 rle with old method : 0.06457328796386719 length of segment : 286 time for calcul the mask position with numpy : 0.00010347366333007812 nb_pixel_total : 2914 time to create 1 rle with old method : 0.003484964370727539 length of segment : 103 time for calcul the mask position with numpy : 0.028653383255004883 nb_pixel_total : 1342761 time to create 1 rle with new method : 0.09473323822021484 length of segment : 2420 time for calcul the mask position with numpy : 0.0007128715515136719 nb_pixel_total : 32491 time to create 1 rle with old method : 0.04074883460998535 length of segment : 261 time for calcul the mask position with numpy : 0.0005784034729003906 nb_pixel_total : 26456 time to create 1 rle with old method : 0.03493046760559082 length of segment : 295 time for calcul the mask position with numpy : 0.0010521411895751953 nb_pixel_total : 52032 time to create 1 rle with old method : 0.06647539138793945 length of segment : 340 time for calcul the mask position with numpy : 0.00048160552978515625 nb_pixel_total : 19477 time to create 1 rle with old method : 0.024176836013793945 length of segment : 182 time for calcul the mask position with numpy : 0.0008180141448974609 nb_pixel_total : 48413 time to create 1 rle with old method : 0.06363463401794434 length of segment : 369 time for calcul the mask position with numpy : 0.0007255077362060547 nb_pixel_total : 37672 time to create 1 rle with old method : 0.04938340187072754 length of segment : 255 time for calcul the mask position with numpy : 0.0008602142333984375 nb_pixel_total : 10564 time to create 1 rle with old method : 0.014068603515625 length of segment : 310 time for calcul the mask position with numpy : 0.0044667720794677734 nb_pixel_total : 315418 time to create 1 rle with new method : 0.010936498641967773 length of segment : 456 time for calcul the mask position with numpy : 0.00011372566223144531 nb_pixel_total : 756 time to create 1 rle with old method : 0.0012249946594238281 length of segment : 34 time for calcul the mask position with numpy : 0.0005657672882080078 nb_pixel_total : 31210 time to create 1 rle with old method : 0.040914297103881836 length of segment : 201 time for calcul the mask position with numpy : 0.004179954528808594 nb_pixel_total : 290600 time to create 1 rle with new method : 0.009404897689819336 length of segment : 385 time for calcul the mask position with numpy : 0.0038537979125976562 nb_pixel_total : 228173 time to create 1 rle with new method : 0.010907173156738281 length of segment : 798 time for calcul the mask position with numpy : 0.05764007568359375 nb_pixel_total : 3476384 time to create 1 rle with new method : 0.1344296932220459 length of segment : 4837 time for calcul the mask position with numpy : 0.00010967254638671875 nb_pixel_total : 2062 time to create 1 rle with old method : 0.0025098323822021484 length of segment : 62 time for calcul the mask position with numpy : 0.0012950897216796875 nb_pixel_total : 116023 time to create 1 rle with old method : 0.12004566192626953 length of segment : 587 time for calcul the mask position with numpy : 0.017136812210083008 nb_pixel_total : 1511158 time to create 1 rle with new method : 0.042398929595947266 length of segment : 2445 time for calcul the mask position with numpy : 0.011864900588989258 nb_pixel_total : 880156 time to create 1 rle with new method : 0.030999183654785156 length of segment : 1362 time for calcul the mask position with numpy : 0.00023603439331054688 nb_pixel_total : 3897 time to create 1 rle with old method : 0.00436091423034668 length of segment : 64 time for calcul the mask position with numpy : 0.00011777877807617188 nb_pixel_total : 1398 time to create 1 rle with old method : 0.0016257762908935547 length of segment : 38 time for calcul the mask position with numpy : 0.033447980880737305 nb_pixel_total : 1737536 time to create 1 rle with new method : 0.05609774589538574 length of segment : 1900 time for calcul the mask position with numpy : 0.0002918243408203125 nb_pixel_total : 4004 time to create 1 rle with old method : 0.004458904266357422 length of segment : 66 time for calcul the mask position with numpy : 0.0003657341003417969 nb_pixel_total : 10985 time to create 1 rle with old method : 0.012608766555786133 length of segment : 97 time for calcul the mask position with numpy : 0.00014090538024902344 nb_pixel_total : 2319 time to create 1 rle with old method : 0.002551555633544922 length of segment : 60 time for calcul the mask position with numpy : 0.003986835479736328 nb_pixel_total : 230844 time to create 1 rle with new method : 0.007241249084472656 length of segment : 557 time for calcul the mask position with numpy : 0.00027370452880859375 nb_pixel_total : 10909 time to create 1 rle with old method : 0.017848491668701172 length of segment : 135 time for calcul the mask position with numpy : 0.00010228157043457031 nb_pixel_total : 1251 time to create 1 rle with old method : 0.002254009246826172 length of segment : 53 time for calcul the mask position with numpy : 0.0002961158752441406 nb_pixel_total : 3653 time to create 1 rle with old method : 0.0060880184173583984 length of segment : 79 time for calcul the mask position with numpy : 0.0018100738525390625 nb_pixel_total : 47274 time to create 1 rle with old method : 0.05046725273132324 length of segment : 409 time for calcul the mask position with numpy : 9.465217590332031e-05 nb_pixel_total : 2443 time to create 1 rle with old method : 0.002914905548095703 length of segment : 55 time for calcul the mask position with numpy : 0.005552768707275391 nb_pixel_total : 397101 time to create 1 rle with new method : 0.012111902236938477 length of segment : 616 time for calcul the mask position with numpy : 0.0006227493286132812 nb_pixel_total : 15933 time to create 1 rle with old method : 0.017460346221923828 length of segment : 215 time for calcul the mask position with numpy : 0.00014209747314453125 nb_pixel_total : 1945 time to create 1 rle with old method : 0.002299785614013672 length of segment : 67 time for calcul the mask position with numpy : 0.0004875659942626953 nb_pixel_total : 23707 time to create 1 rle with old method : 0.025028467178344727 length of segment : 274 time for calcul the mask position with numpy : 0.0002949237823486328 nb_pixel_total : 14996 time to create 1 rle with old method : 0.01682138442993164 length of segment : 212 time for calcul the mask position with numpy : 0.0012044906616210938 nb_pixel_total : 51587 time to create 1 rle with old method : 0.056388139724731445 length of segment : 377 time for calcul the mask position with numpy : 0.00823974609375 nb_pixel_total : 509001 time to create 1 rle with new method : 0.021010160446166992 length of segment : 653 time for calcul the mask position with numpy : 0.00022912025451660156 nb_pixel_total : 4314 time to create 1 rle with old method : 0.005282163619995117 length of segment : 72 time for calcul the mask position with numpy : 0.0001976490020751953 nb_pixel_total : 3216 time to create 1 rle with old method : 0.0038766860961914062 length of segment : 70 time for calcul the mask position with numpy : 0.0004904270172119141 nb_pixel_total : 11898 time to create 1 rle with old method : 0.01266169548034668 length of segment : 155 time for calcul the mask position with numpy : 0.001965761184692383 nb_pixel_total : 7855 time to create 1 rle with old method : 0.008769989013671875 length of segment : 148 time for calcul the mask position with numpy : 0.0006933212280273438 nb_pixel_total : 22982 time to create 1 rle with old method : 0.025249004364013672 length of segment : 186 time for calcul the mask position with numpy : 0.0014476776123046875 nb_pixel_total : 37356 time to create 1 rle with old method : 0.0391848087310791 length of segment : 267 time for calcul the mask position with numpy : 0.0009915828704833984 nb_pixel_total : 24713 time to create 1 rle with old method : 0.02742767333984375 length of segment : 184 time for calcul the mask position with numpy : 0.00024390220642089844 nb_pixel_total : 3768 time to create 1 rle with old method : 0.004355192184448242 length of segment : 89 time for calcul the mask position with numpy : 0.0002713203430175781 nb_pixel_total : 5135 time to create 1 rle with old method : 0.005661487579345703 length of segment : 88 time for calcul the mask position with numpy : 0.0013439655303955078 nb_pixel_total : 33220 time to create 1 rle with old method : 0.036380767822265625 length of segment : 338 time for calcul the mask position with numpy : 0.004828214645385742 nb_pixel_total : 58201 time to create 1 rle with old method : 0.06319212913513184 length of segment : 585 time for calcul the mask position with numpy : 0.003113269805908203 nb_pixel_total : 89299 time to create 1 rle with old method : 0.09438037872314453 length of segment : 425 time for calcul the mask position with numpy : 0.001928567886352539 nb_pixel_total : 52580 time to create 1 rle with old method : 0.05343914031982422 length of segment : 394 time for calcul the mask position with numpy : 0.0023987293243408203 nb_pixel_total : 79734 time to create 1 rle with old method : 0.08339571952819824 length of segment : 515 time for calcul the mask position with numpy : 0.0004069805145263672 nb_pixel_total : 8639 time to create 1 rle with old method : 0.008842706680297852 length of segment : 162 time for calcul the mask position with numpy : 0.00582432746887207 nb_pixel_total : 209786 time to create 1 rle with new method : 0.012222051620483398 length of segment : 615 time for calcul the mask position with numpy : 0.0010869503021240234 nb_pixel_total : 29334 time to create 1 rle with old method : 0.03198432922363281 length of segment : 309 time for calcul the mask position with numpy : 0.00022411346435546875 nb_pixel_total : 11303 time to create 1 rle with old method : 0.011970758438110352 length of segment : 137 time for calcul the mask position with numpy : 0.0005209445953369141 nb_pixel_total : 12624 time to create 1 rle with old method : 0.013486862182617188 length of segment : 195 time for calcul the mask position with numpy : 0.00043487548828125 nb_pixel_total : 23915 time to create 1 rle with old method : 0.025104999542236328 length of segment : 186 time for calcul the mask position with numpy : 0.00024771690368652344 nb_pixel_total : 4318 time to create 1 rle with old method : 0.00492095947265625 length of segment : 37 time for calcul the mask position with numpy : 0.001619577407836914 nb_pixel_total : 43592 time to create 1 rle with old method : 0.0456385612487793 length of segment : 279 time for calcul the mask position with numpy : 0.0003211498260498047 nb_pixel_total : 6221 time to create 1 rle with old method : 0.006773233413696289 length of segment : 121 time for calcul the mask position with numpy : 0.0013532638549804688 nb_pixel_total : 57898 time to create 1 rle with old method : 0.06116795539855957 length of segment : 483 time for calcul the mask position with numpy : 0.0011146068572998047 nb_pixel_total : 43050 time to create 1 rle with old method : 0.0445253849029541 length of segment : 290 time for calcul the mask position with numpy : 0.00016117095947265625 nb_pixel_total : 2115 time to create 1 rle with old method : 0.002212047576904297 length of segment : 76 time for calcul the mask position with numpy : 0.0015728473663330078 nb_pixel_total : 46107 time to create 1 rle with old method : 0.047484397888183594 length of segment : 371 time for calcul the mask position with numpy : 0.003263235092163086 nb_pixel_total : 129522 time to create 1 rle with old method : 0.13198304176330566 length of segment : 524 time for calcul the mask position with numpy : 0.005221128463745117 nb_pixel_total : 229665 time to create 1 rle with new method : 0.007979869842529297 length of segment : 495 time for calcul the mask position with numpy : 0.0006604194641113281 nb_pixel_total : 20958 time to create 1 rle with old method : 0.021711111068725586 length of segment : 186 time for calcul the mask position with numpy : 0.0032851696014404297 nb_pixel_total : 132681 time to create 1 rle with old method : 0.13558077812194824 length of segment : 655 time for calcul the mask position with numpy : 0.003792285919189453 nb_pixel_total : 115921 time to create 1 rle with old method : 0.12162971496582031 length of segment : 486 time for calcul the mask position with numpy : 0.0005242824554443359 nb_pixel_total : 16897 time to create 1 rle with old method : 0.018246173858642578 length of segment : 162 time for calcul the mask position with numpy : 0.0008080005645751953 nb_pixel_total : 43430 time to create 1 rle with old method : 0.04753375053405762 length of segment : 352 time for calcul the mask position with numpy : 9.942054748535156e-05 nb_pixel_total : 1670 time to create 1 rle with old method : 0.0018565654754638672 length of segment : 41 time for calcul the mask position with numpy : 0.0003619194030761719 nb_pixel_total : 18767 time to create 1 rle with old method : 0.020172834396362305 length of segment : 182 time for calcul the mask position with numpy : 0.00041174888610839844 nb_pixel_total : 20588 time to create 1 rle with old method : 0.021471738815307617 length of segment : 191 time for calcul the mask position with numpy : 0.0006260871887207031 nb_pixel_total : 42988 time to create 1 rle with old method : 0.044496774673461914 length of segment : 353 time for calcul the mask position with numpy : 0.002007007598876953 nb_pixel_total : 116645 time to create 1 rle with old method : 0.11694812774658203 length of segment : 488 time for calcul the mask position with numpy : 0.0018131732940673828 nb_pixel_total : 71156 time to create 1 rle with old method : 0.0767669677734375 length of segment : 372 time for calcul the mask position with numpy : 0.0007677078247070312 nb_pixel_total : 23264 time to create 1 rle with old method : 0.027491092681884766 length of segment : 180 time for calcul the mask position with numpy : 0.0003788471221923828 nb_pixel_total : 11442 time to create 1 rle with old method : 0.012152910232543945 length of segment : 145 time for calcul the mask position with numpy : 9.846687316894531e-05 nb_pixel_total : 1094 time to create 1 rle with old method : 0.0012068748474121094 length of segment : 41 time for calcul the mask position with numpy : 0.00029397010803222656 nb_pixel_total : 6439 time to create 1 rle with old method : 0.0071179866790771484 length of segment : 123 time for calcul the mask position with numpy : 0.0013949871063232422 nb_pixel_total : 69108 time to create 1 rle with old method : 0.07104945182800293 length of segment : 387 time for calcul the mask position with numpy : 0.0015015602111816406 nb_pixel_total : 57635 time to create 1 rle with old method : 0.05803799629211426 length of segment : 440 time for calcul the mask position with numpy : 0.002457141876220703 nb_pixel_total : 74012 time to create 1 rle with old method : 0.0780019760131836 length of segment : 537 time for calcul the mask position with numpy : 0.0025205612182617188 nb_pixel_total : 60973 time to create 1 rle with old method : 0.06377172470092773 length of segment : 689 time for calcul the mask position with numpy : 0.00030040740966796875 nb_pixel_total : 12896 time to create 1 rle with old method : 0.01411294937133789 length of segment : 115 time for calcul the mask position with numpy : 0.0012545585632324219 nb_pixel_total : 50793 time to create 1 rle with old method : 0.05182623863220215 length of segment : 374 time for calcul the mask position with numpy : 0.00029206275939941406 nb_pixel_total : 12121 time to create 1 rle with old method : 0.013291358947753906 length of segment : 180 time for calcul the mask position with numpy : 0.0024919509887695312 nb_pixel_total : 120080 time to create 1 rle with old method : 0.12656903266906738 length of segment : 677 time for calcul the mask position with numpy : 0.0003674030303955078 nb_pixel_total : 10634 time to create 1 rle with old method : 0.011364936828613281 length of segment : 136 time for calcul the mask position with numpy : 0.00030303001403808594 nb_pixel_total : 7307 time to create 1 rle with old method : 0.008206367492675781 length of segment : 91 time for calcul the mask position with numpy : 0.0002868175506591797 nb_pixel_total : 4276 time to create 1 rle with old method : 0.0048902034759521484 length of segment : 100 time for calcul the mask position with numpy : 0.000141143798828125 nb_pixel_total : 2043 time to create 1 rle with old method : 0.002271890640258789 length of segment : 61 time for calcul the mask position with numpy : 0.0007088184356689453 nb_pixel_total : 21463 time to create 1 rle with old method : 0.023813962936401367 length of segment : 152 time for calcul the mask position with numpy : 0.00045108795166015625 nb_pixel_total : 13302 time to create 1 rle with old method : 0.014104127883911133 length of segment : 142 time for calcul the mask position with numpy : 0.0006518363952636719 nb_pixel_total : 25368 time to create 1 rle with old method : 0.027348041534423828 length of segment : 129 time for calcul the mask position with numpy : 0.00023674964904785156 nb_pixel_total : 3717 time to create 1 rle with old method : 0.004326820373535156 length of segment : 86 time for calcul the mask position with numpy : 0.00019860267639160156 nb_pixel_total : 1459 time to create 1 rle with old method : 0.0017852783203125 length of segment : 58 time for calcul the mask position with numpy : 0.0003516674041748047 nb_pixel_total : 7519 time to create 1 rle with old method : 0.008951187133789062 length of segment : 102 time for calcul the mask position with numpy : 0.0012362003326416016 nb_pixel_total : 15254 time to create 1 rle with old method : 0.017780303955078125 length of segment : 282 time for calcul the mask position with numpy : 0.0006055831909179688 nb_pixel_total : 18469 time to create 1 rle with old method : 0.02072286605834961 length of segment : 207 time for calcul the mask position with numpy : 0.0016798973083496094 nb_pixel_total : 50557 time to create 1 rle with old method : 0.05399942398071289 length of segment : 298 time for calcul the mask position with numpy : 0.00034499168395996094 nb_pixel_total : 8682 time to create 1 rle with old method : 0.010437726974487305 length of segment : 89 time for calcul the mask position with numpy : 0.00833439826965332 nb_pixel_total : 287412 time to create 1 rle with new method : 0.014182329177856445 length of segment : 662 time for calcul the mask position with numpy : 0.00013637542724609375 nb_pixel_total : 765 time to create 1 rle with old method : 0.0009601116180419922 length of segment : 44 time for calcul the mask position with numpy : 0.004045009613037109 nb_pixel_total : 94554 time to create 1 rle with old method : 0.101837158203125 length of segment : 334 time for calcul the mask position with numpy : 0.0002448558807373047 nb_pixel_total : 2912 time to create 1 rle with old method : 0.003370523452758789 length of segment : 92 time for calcul the mask position with numpy : 0.0001380443572998047 nb_pixel_total : 1222 time to create 1 rle with old method : 0.0016148090362548828 length of segment : 34 time for calcul the mask position with numpy : 0.00020456314086914062 nb_pixel_total : 5126 time to create 1 rle with old method : 0.006191730499267578 length of segment : 81 time for calcul the mask position with numpy : 0.0002760887145996094 nb_pixel_total : 4481 time to create 1 rle with old method : 0.00511479377746582 length of segment : 115 time for calcul the mask position with numpy : 0.06081557273864746 nb_pixel_total : 3372617 time to create 1 rle with new method : 0.12006044387817383 length of segment : 3557 time for calcul the mask position with numpy : 0.002279996871948242 nb_pixel_total : 188797 time to create 1 rle with new method : 0.0053136348724365234 length of segment : 576 time for calcul the mask position with numpy : 0.003230571746826172 nb_pixel_total : 258921 time to create 1 rle with new method : 0.006722927093505859 length of segment : 515 time for calcul the mask position with numpy : 0.0027115345001220703 nb_pixel_total : 263629 time to create 1 rle with new method : 0.005760908126831055 length of segment : 506 time for calcul the mask position with numpy : 0.0008566379547119141 nb_pixel_total : 53780 time to create 1 rle with old method : 0.05919647216796875 length of segment : 266 time for calcul the mask position with numpy : 0.01143026351928711 nb_pixel_total : 725461 time to create 1 rle with new method : 0.029410839080810547 length of segment : 1425 time for calcul the mask position with numpy : 0.003889322280883789 nb_pixel_total : 145620 time to create 1 rle with old method : 0.15761280059814453 length of segment : 374 time for calcul the mask position with numpy : 0.004833221435546875 nb_pixel_total : 207672 time to create 1 rle with new method : 0.006173372268676758 length of segment : 465 time for calcul the mask position with numpy : 0.0048885345458984375 nb_pixel_total : 187703 time to create 1 rle with new method : 0.00644683837890625 length of segment : 477 time for calcul the mask position with numpy : 0.0034804344177246094 nb_pixel_total : 88381 time to create 1 rle with old method : 0.10476064682006836 length of segment : 544 time for calcul the mask position with numpy : 0.0006918907165527344 nb_pixel_total : 18139 time to create 1 rle with old method : 0.019828319549560547 length of segment : 164 time for calcul the mask position with numpy : 0.0003540515899658203 nb_pixel_total : 6236 time to create 1 rle with old method : 0.0071032047271728516 length of segment : 110 time for calcul the mask position with numpy : 0.0010530948638916016 nb_pixel_total : 29393 time to create 1 rle with old method : 0.03162097930908203 length of segment : 232 time for calcul the mask position with numpy : 0.0001049041748046875 nb_pixel_total : 2974 time to create 1 rle with old method : 0.0034101009368896484 length of segment : 63 time for calcul the mask position with numpy : 0.0007953643798828125 nb_pixel_total : 17094 time to create 1 rle with old method : 0.01839733123779297 length of segment : 280 time for calcul the mask position with numpy : 0.0010943412780761719 nb_pixel_total : 13846 time to create 1 rle with old method : 0.01575756072998047 length of segment : 226 time for calcul the mask position with numpy : 0.0011670589447021484 nb_pixel_total : 38862 time to create 1 rle with old method : 0.04175519943237305 length of segment : 246 time for calcul the mask position with numpy : 0.0018625259399414062 nb_pixel_total : 88535 time to create 1 rle with old method : 0.09389662742614746 length of segment : 287 time for calcul the mask position with numpy : 0.0007219314575195312 nb_pixel_total : 18045 time to create 1 rle with old method : 0.01963019371032715 length of segment : 209 time for calcul the mask position with numpy : 0.00027060508728027344 nb_pixel_total : 2826 time to create 1 rle with old method : 0.003260374069213867 length of segment : 68 time for calcul the mask position with numpy : 0.0029528141021728516 nb_pixel_total : 189373 time to create 1 rle with new method : 0.007926702499389648 length of segment : 480 time for calcul the mask position with numpy : 0.0015113353729248047 nb_pixel_total : 45809 time to create 1 rle with old method : 0.04787611961364746 length of segment : 312 time for calcul the mask position with numpy : 0.007233619689941406 nb_pixel_total : 192398 time to create 1 rle with new method : 0.011744260787963867 length of segment : 721 time for calcul the mask position with numpy : 0.02926921844482422 nb_pixel_total : 1034135 time to create 1 rle with new method : 0.05551791191101074 length of segment : 1772 time for calcul the mask position with numpy : 0.0004684925079345703 nb_pixel_total : 15970 time to create 1 rle with old method : 0.017399072647094727 length of segment : 182 time for calcul the mask position with numpy : 0.00013399124145507812 nb_pixel_total : 6373 time to create 1 rle with old method : 0.007244110107421875 length of segment : 89 time for calcul the mask position with numpy : 0.0002033710479736328 nb_pixel_total : 10997 time to create 1 rle with old method : 0.012247800827026367 length of segment : 115 time spent for convertir_results : 15.830257177352905 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 158 chid ids of type : 3663 Number RLEs to save : 59956 save missing photos in datou_result : time spend for datou_step_exec : 48.15827250480652 time spend to save output : 4.539092779159546 total time spend for step 1 : 52.697365283966064 step2:crop_condition Wed Feb 5 11:55:23 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 158 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 13 About to insert : list_path_to_insert length 13 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 46 About to insert : list_path_to_insert length 46 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1334704064, 1334704062, 1334704060, 1334703877, 1334703874, 1334703870, 1334703864, 1334703861, 1334703859, 1334703710] Looping around the photos to save general results len do output : 84 /-3655846257Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846268Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846327Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846338Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846365Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846354Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846361Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846359Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846381Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846271Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846269Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846283Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846309Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846305Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846306Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846318Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846353Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846347Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846366Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846362Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846379Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846403Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846266Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846279Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846328Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846325Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846321Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846342Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846348Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846346Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846408Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846265Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846267Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846260Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846273Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846270Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846262Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846275Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846285Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846281Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846286Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846288Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846293Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846302Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846313Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846308Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846311Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846324Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846330Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846331Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846336Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846329Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846320Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846326Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846341Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846339Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846343Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846351Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846364Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846360Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846373Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846383Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846392Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846391Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846390Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846389Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846400Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846402Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846394Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846397Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846409Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846396Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846395Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846405Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846398Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846404Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846413Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846282Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846300Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846299Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846303Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846344Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846371Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655846284Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704064', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704062', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704060', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703877', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703874', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703870', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703864', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703861', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703859', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703710', None, None, None, None, None, '2551122') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 262 time used for this insertion : 0.03168773651123047 save_final save missing photos in datou_result : time spend for datou_step_exec : 35.136924505233765 time spend to save output : 0.03389263153076172 total time spend for step 2 : 35.170817136764526 step3:thcl Wed Feb 5 11:55:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.006758213043212891 time to convert the images to numpy array : 0.020994901657104492 time to import caffe and check if the image exist : 0.009363651275634766 time to convert the images to numpy array : 0.06430888175964355 time to import caffe and check if the image exist : 0.012299299240112305 time to convert the images to numpy array : 0.09117674827575684 time to import caffe and check if the image exist : 0.011762857437133789 time to convert the images to numpy array : 0.09884381294250488 time to import caffe and check if the image exist : 0.007098674774169922 time to convert the images to numpy array : 0.109283447265625 time to import caffe and check if the image exist : 0.007448434829711914 time to convert the images to numpy array : 0.10785222053527832 time to import caffe and check if the image exist : 0.010614633560180664 time to convert the images to numpy array : 0.11416435241699219 time to import caffe and check if the image exist : 0.010526418685913086 time to convert the images to numpy array : 0.12352275848388672 time to import caffe and check if the image exist : 0.010133028030395508 time to convert the images to numpy array : 0.13164377212524414 time to import caffe and check if the image exist : 0.010807037353515625 time to convert the images to numpy array : 0.13161540031433105 total time to convert the images to numpy array : 0.34619593620300293 list photo_ids error: [] list photo_ids correct : [-3655846344, -3655846371, -3655846284, -3655846395, -3655846405, -3655846398, -3655846404, -3655846413, -3655846282, -3655846300, -3655846299, -3655846303, -3655846271, -3655846269, -3655846283, -3655846309, -3655846305, -3655846306, -3655846318, -3655846353, -3655846347, -3655846341, -3655846339, -3655846343, -3655846351, -3655846364, -3655846360, -3655846373, -3655846383, -3655846392, -3655846257, -3655846268, -3655846327, -3655846338, -3655846365, -3655846354, -3655846361, -3655846359, -3655846381, -3655846366, -3655846362, -3655846379, -3655846403, -3655846266, -3655846279, -3655846328, -3655846325, -3655846321, -3655846308, -3655846311, -3655846324, -3655846330, -3655846331, -3655846336, -3655846329, -3655846320, -3655846326, -3655846262, -3655846275, -3655846285, -3655846281, -3655846286, -3655846288, -3655846293, -3655846302, -3655846313, -3655846391, -3655846390, -3655846389, -3655846400, -3655846402, -3655846394, -3655846397, -3655846409, -3655846396, -3655846342, -3655846348, -3655846346, -3655846408, -3655846265, -3655846267, -3655846260, -3655846273, -3655846270] number of photos to traite : 84 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6997 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6778 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.0070443153381348 time used to do the prediction : 0.22869658470153809 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6776 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6776 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.9230697154998779 time used to do the prediction : 0.1625807285308838 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.5735626220703125e-05 save missing photos in datou_result : time spend for datou_step_exec : 9.263460874557495 time spend to save output : 0.0005757808685302734 total time spend for step 3 : 9.264036655426025 step4:merge_mask_thcl_custom Wed Feb 5 11:56:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 158 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1334704064, 1334704062, 1334704060, 1334703877, 1334703874, 1334703870, 1334703864, 1334703861, 1334703859, 1334703710] Looping around the photos to save general results len do output : 10 /1334704064Didn't retrieve data .Didn't retrieve data . /1334704062Didn't retrieve data .Didn't retrieve data . /1334704060Didn't retrieve data .Didn't retrieve data . /1334703877Didn't retrieve data .Didn't retrieve data . /1334703874Didn't retrieve data .Didn't retrieve data . /1334703870Didn't retrieve data .Didn't retrieve data . /1334703864Didn't retrieve data .Didn't retrieve data . /1334703861Didn't retrieve data .Didn't retrieve data . /1334703859Didn't retrieve data .Didn't retrieve data . /1334703710Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704064', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704062', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704060', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703877', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703874', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703870', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703864', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703861', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703859', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703710', None, None, None, None, None, '2551122') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013892889022827148 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.13105249404907227 time spend to save output : 0.014436006546020508 total time spend for step 4 : 0.14548850059509277 step5:rle_unique_nms_with_priority Wed Feb 5 11:56:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 17 nb_hashtags : 8 time to prepare the origin masks : 6.876442193984985 time for calcul the mask position with numpy : 0.28067731857299805 nb_pixel_total : 6044688 time to create 1 rle with new method : 0.7082531452178955 time for calcul the mask position with numpy : 0.022962570190429688 nb_pixel_total : 38263 time to create 1 rle with old method : 0.05000448226928711 time for calcul the mask position with numpy : 0.020746707916259766 nb_pixel_total : 52421 time to create 1 rle with old method : 0.05379652976989746 time for calcul the mask position with numpy : 0.0213015079498291 nb_pixel_total : 84393 time to create 1 rle with old method : 0.09262609481811523 time for calcul the mask position with numpy : 0.021088361740112305 nb_pixel_total : 62577 time to create 1 rle with old method : 0.06466197967529297 time for calcul the mask position with numpy : 0.020520687103271484 nb_pixel_total : 83078 time to create 1 rle with old method : 0.0830390453338623 time for calcul the mask position with numpy : 0.02044081687927246 nb_pixel_total : 72691 time to create 1 rle with old method : 0.07261991500854492 time for calcul the mask position with numpy : 0.022125244140625 nb_pixel_total : 386463 time to create 1 rle with new method : 0.3594040870666504 time for calcul the mask position with numpy : 0.02069234848022461 nb_pixel_total : 5985 time to create 1 rle with old method : 0.006224155426025391 time for calcul the mask position with numpy : 0.020011186599731445 nb_pixel_total : 171 time to create 1 rle with old method : 0.0003800392150878906 time for calcul the mask position with numpy : 0.023412227630615234 nb_pixel_total : 3132 time to create 1 rle with old method : 0.003561258316040039 time for calcul the mask position with numpy : 0.020415306091308594 nb_pixel_total : 47732 time to create 1 rle with old method : 0.04806113243103027 time for calcul the mask position with numpy : 0.024740934371948242 nb_pixel_total : 32161 time to create 1 rle with old method : 0.03327655792236328 time for calcul the mask position with numpy : 0.020258665084838867 nb_pixel_total : 12899 time to create 1 rle with old method : 0.013273477554321289 time for calcul the mask position with numpy : 0.021127939224243164 nb_pixel_total : 10028 time to create 1 rle with old method : 0.010598182678222656 time for calcul the mask position with numpy : 0.02992415428161621 nb_pixel_total : 7966 time to create 1 rle with old method : 0.008507728576660156 time for calcul the mask position with numpy : 0.03247380256652832 nb_pixel_total : 29810 time to create 1 rle with old method : 0.029781103134155273 time for calcul the mask position with numpy : 0.033188581466674805 nb_pixel_total : 75782 time to create 1 rle with old method : 0.0737149715423584 create new chi : 2.4398727416992188 time to delete rle : 0.014023303985595703 batch 1 Loaded 18 chid ids of type : 3726 Number RLEs to save : 11471 TO DO : save crop sub photo not yet done ! save time : 0.6377482414245605 nb_obj : 15 nb_hashtags : 5 time to prepare the origin masks : 5.274502515792847 time for calcul the mask position with numpy : 0.08666849136352539 nb_pixel_total : 4951733 time to create 1 rle with new method : 1.4771592617034912 time for calcul the mask position with numpy : 0.02200174331665039 nb_pixel_total : 7286 time to create 1 rle with old method : 0.008424997329711914 time for calcul the mask position with numpy : 0.02241206169128418 nb_pixel_total : 30975 time to create 1 rle with old method : 0.03438091278076172 time for calcul the mask position with numpy : 0.023624897003173828 nb_pixel_total : 19259 time to create 1 rle with old method : 0.026633739471435547 time for calcul the mask position with numpy : 0.023860931396484375 nb_pixel_total : 55201 time to create 1 rle with old method : 0.06578755378723145 time for calcul the mask position with numpy : 0.023124217987060547 nb_pixel_total : 133650 time to create 1 rle with old method : 0.14657974243164062 time for calcul the mask position with numpy : 0.020818233489990234 nb_pixel_total : 51900 time to create 1 rle with old method : 0.055680036544799805 time for calcul the mask position with numpy : 0.021868467330932617 nb_pixel_total : 289983 time to create 1 rle with new method : 0.5677738189697266 time for calcul the mask position with numpy : 0.022631168365478516 nb_pixel_total : 750 time to create 1 rle with old method : 0.0008785724639892578 time for calcul the mask position with numpy : 0.022623062133789062 nb_pixel_total : 32415 time to create 1 rle with old method : 0.0338137149810791 time for calcul the mask position with numpy : 0.021019458770751953 nb_pixel_total : 26449 time to create 1 rle with old method : 0.02790665626525879 time for calcul the mask position with numpy : 0.03249549865722656 nb_pixel_total : 1338898 time to create 1 rle with new method : 0.3036010265350342 time for calcul the mask position with numpy : 0.02150869369506836 nb_pixel_total : 2889 time to create 1 rle with old method : 0.003299236297607422 time for calcul the mask position with numpy : 0.031087636947631836 nb_pixel_total : 60743 time to create 1 rle with old method : 0.06430172920227051 time for calcul the mask position with numpy : 0.022209644317626953 nb_pixel_total : 37612 time to create 1 rle with old method : 0.03943037986755371 time for calcul the mask position with numpy : 0.020852088928222656 nb_pixel_total : 10497 time to create 1 rle with old method : 0.0116119384765625 create new chi : 3.374985694885254 time to delete rle : 0.0012943744659423828 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 14523 TO DO : save crop sub photo not yet done ! save time : 0.8739285469055176 nb_obj : 5 nb_hashtags : 2 time to prepare the origin masks : 1.5266492366790771 time for calcul the mask position with numpy : 0.04200887680053711 nb_pixel_total : 2939405 time to create 1 rle with new method : 0.13887572288513184 time for calcul the mask position with numpy : 0.020726442337036133 nb_pixel_total : 164897 time to create 1 rle with new method : 0.13378620147705078 time for calcul the mask position with numpy : 0.024431943893432617 nb_pixel_total : 400984 time to create 1 rle with new method : 0.13518762588500977 time for calcul the mask position with numpy : 0.020041227340698242 nb_pixel_total : 2047 time to create 1 rle with old method : 0.0022115707397460938 time for calcul the mask position with numpy : 0.020254850387573242 nb_pixel_total : 80312 time to create 1 rle with old method : 0.08165788650512695 time for calcul the mask position with numpy : 0.04660296440124512 nb_pixel_total : 3462595 time to create 1 rle with new method : 0.13029861450195312 create new chi : 0.854949951171875 time to delete rle : 0.0012989044189453125 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 17162 TO DO : save crop sub photo not yet done ! save time : 0.9142308235168457 nb_obj : 19 nb_hashtags : 7 time to prepare the origin masks : 7.187901973724365 time for calcul the mask position with numpy : 0.08429169654846191 nb_pixel_total : 4738632 time to create 1 rle with new method : 0.8562479019165039 time for calcul the mask position with numpy : 0.0221099853515625 nb_pixel_total : 15904 time to create 1 rle with old method : 0.016878843307495117 time for calcul the mask position with numpy : 0.020310163497924805 nb_pixel_total : 51428 time to create 1 rle with old method : 0.05231213569641113 time for calcul the mask position with numpy : 0.020330429077148438 nb_pixel_total : 3638 time to create 1 rle with old method : 0.004105567932128906 time for calcul the mask position with numpy : 0.02397465705871582 nb_pixel_total : 10934 time to create 1 rle with old method : 0.011380434036254883 time for calcul the mask position with numpy : 0.01985025405883789 nb_pixel_total : 3978 time to create 1 rle with old method : 0.003973484039306641 time for calcul the mask position with numpy : 0.020099163055419922 nb_pixel_total : 4255 time to create 1 rle with old method : 0.004759073257446289 time for calcul the mask position with numpy : 0.022077083587646484 nb_pixel_total : 58306 time to create 1 rle with old method : 0.09085941314697266 time for calcul the mask position with numpy : 0.020577430725097656 nb_pixel_total : 1937 time to create 1 rle with old method : 0.002048492431640625 time for calcul the mask position with numpy : 0.02048945426940918 nb_pixel_total : 8341 time to create 1 rle with old method : 0.009084463119506836 time for calcul the mask position with numpy : 0.02685689926147461 nb_pixel_total : 2420 time to create 1 rle with old method : 0.002756357192993164 time for calcul the mask position with numpy : 0.03015875816345215 nb_pixel_total : 14135 time to create 1 rle with old method : 0.015446662902832031 time for calcul the mask position with numpy : 0.02048492431640625 nb_pixel_total : 47185 time to create 1 rle with old method : 0.04933953285217285 time for calcul the mask position with numpy : 0.021811485290527344 nb_pixel_total : 113123 time to create 1 rle with old method : 0.11808204650878906 time for calcul the mask position with numpy : 0.02198195457458496 nb_pixel_total : 1236 time to create 1 rle with old method : 0.0014688968658447266 time for calcul the mask position with numpy : 0.02156662940979004 nb_pixel_total : 2310 time to create 1 rle with old method : 0.002614736557006836 time for calcul the mask position with numpy : 0.04680204391479492 nb_pixel_total : 1736563 time to create 1 rle with new method : 0.14628314971923828 time for calcul the mask position with numpy : 0.03758692741394043 nb_pixel_total : 230648 time to create 1 rle with new method : 0.13525938987731934 time for calcul the mask position with numpy : 0.032881736755371094 nb_pixel_total : 1396 time to create 1 rle with old method : 0.001508951187133789 time for calcul the mask position with numpy : 0.03306150436401367 nb_pixel_total : 3871 time to create 1 rle with old method : 0.004220485687255859 create new chi : 2.1467905044555664 time to delete rle : 0.001512765884399414 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 11618 TO DO : save crop sub photo not yet done ! save time : 0.7059590816497803 nb_obj : 19 nb_hashtags : 6 time to prepare the origin masks : 7.119101047515869 time for calcul the mask position with numpy : 0.35135912895202637 nb_pixel_total : 6370754 time to create 1 rle with new method : 0.7995963096618652 time for calcul the mask position with numpy : 0.020576000213623047 nb_pixel_total : 89344 time to create 1 rle with old method : 0.09158205986022949 time for calcul the mask position with numpy : 0.020966291427612305 nb_pixel_total : 37225 time to create 1 rle with old method : 0.03769087791442871 time for calcul the mask position with numpy : 0.019014596939086914 nb_pixel_total : 4266 time to create 1 rle with old method : 0.0044155120849609375 time for calcul the mask position with numpy : 0.01975250244140625 nb_pixel_total : 12578 time to create 1 rle with old method : 0.013512134552001953 time for calcul the mask position with numpy : 0.02193593978881836 nb_pixel_total : 209213 time to create 1 rle with new method : 0.2950742244720459 time for calcul the mask position with numpy : 0.019845008850097656 nb_pixel_total : 30181 time to create 1 rle with old method : 0.03218865394592285 time for calcul the mask position with numpy : 0.019281387329101562 nb_pixel_total : 8585 time to create 1 rle with old method : 0.009386777877807617 time for calcul the mask position with numpy : 0.019567489624023438 nb_pixel_total : 24664 time to create 1 rle with old method : 0.025138139724731445 time for calcul the mask position with numpy : 0.01913166046142578 nb_pixel_total : 137 time to create 1 rle with old method : 0.00024008750915527344 time for calcul the mask position with numpy : 0.018772602081298828 nb_pixel_total : 5134 time to create 1 rle with old method : 0.0052337646484375 time for calcul the mask position with numpy : 0.018674373626708984 nb_pixel_total : 3730 time to create 1 rle with old method : 0.003804445266723633 time for calcul the mask position with numpy : 0.019897937774658203 nb_pixel_total : 52473 time to create 1 rle with old method : 0.053682804107666016 time for calcul the mask position with numpy : 0.01984572410583496 nb_pixel_total : 23282 time to create 1 rle with old method : 0.02310013771057129 time for calcul the mask position with numpy : 0.019960403442382812 nb_pixel_total : 11874 time to create 1 rle with old method : 0.011787652969360352 time for calcul the mask position with numpy : 0.02027106285095215 nb_pixel_total : 32884 time to create 1 rle with old method : 0.03391456604003906 time for calcul the mask position with numpy : 0.02086353302001953 nb_pixel_total : 3197 time to create 1 rle with old method : 0.0036630630493164062 time for calcul the mask position with numpy : 0.021591901779174805 nb_pixel_total : 29193 time to create 1 rle with old method : 0.030377864837646484 time for calcul the mask position with numpy : 0.022272109985351562 nb_pixel_total : 43538 time to create 1 rle with old method : 0.04515528678894043 time for calcul the mask position with numpy : 0.020996809005737305 nb_pixel_total : 57988 time to create 1 rle with old method : 0.06089377403259277 create new chi : 2.3632359504699707 time to delete rle : 0.0010483264923095703 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 11738 TO DO : save crop sub photo not yet done ! save time : 0.6417827606201172 nb_obj : 15 nb_hashtags : 4 time to prepare the origin masks : 4.034290313720703 time for calcul the mask position with numpy : 0.2709932327270508 nb_pixel_total : 6341040 time to create 1 rle with new method : 0.618922233581543 time for calcul the mask position with numpy : 0.019729137420654297 nb_pixel_total : 18704 time to create 1 rle with old method : 0.019617080688476562 time for calcul the mask position with numpy : 0.01997995376586914 nb_pixel_total : 229478 time to create 1 rle with new method : 0.21903371810913086 time for calcul the mask position with numpy : 0.018919944763183594 nb_pixel_total : 1665 time to create 1 rle with old method : 0.0018129348754882812 time for calcul the mask position with numpy : 0.01887345314025879 nb_pixel_total : 42904 time to create 1 rle with old method : 0.041780948638916016 time for calcul the mask position with numpy : 0.01904749870300293 nb_pixel_total : 2045 time to create 1 rle with old method : 0.0026471614837646484 time for calcul the mask position with numpy : 0.01998305320739746 nb_pixel_total : 116659 time to create 1 rle with old method : 0.11459732055664062 time for calcul the mask position with numpy : 0.018819332122802734 nb_pixel_total : 4071 time to create 1 rle with old method : 0.004342555999755859 time for calcul the mask position with numpy : 0.018538951873779297 nb_pixel_total : 1170 time to create 1 rle with old method : 0.0013310909271240234 time for calcul the mask position with numpy : 0.018975257873535156 nb_pixel_total : 16849 time to create 1 rle with old method : 0.01691436767578125 time for calcul the mask position with numpy : 0.019249916076660156 nb_pixel_total : 2106 time to create 1 rle with old method : 0.0022644996643066406 time for calcul the mask position with numpy : 0.02023601531982422 nb_pixel_total : 20574 time to create 1 rle with old method : 0.020354270935058594 time for calcul the mask position with numpy : 0.019430160522460938 nb_pixel_total : 43322 time to create 1 rle with old method : 0.04336714744567871 time for calcul the mask position with numpy : 0.02845168113708496 nb_pixel_total : 69147 time to create 1 rle with old method : 0.06866335868835449 time for calcul the mask position with numpy : 0.030953645706176758 nb_pixel_total : 6198 time to create 1 rle with old method : 0.0064127445220947266 time for calcul the mask position with numpy : 0.03095245361328125 nb_pixel_total : 134308 time to create 1 rle with old method : 0.1350722312927246 create new chi : 1.9595060348510742 time to delete rle : 0.0008275508880615234 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 9322 TO DO : save crop sub photo not yet done ! save time : 0.5624313354492188 nb_obj : 14 nb_hashtags : 6 time to prepare the origin masks : 3.9753057956695557 time for calcul the mask position with numpy : 0.40209150314331055 nb_pixel_total : 6468371 time to create 1 rle with new method : 0.7073731422424316 time for calcul the mask position with numpy : 0.020588397979736328 nb_pixel_total : 50777 time to create 1 rle with old method : 0.05434870719909668 time for calcul the mask position with numpy : 0.020312070846557617 nb_pixel_total : 119929 time to create 1 rle with old method : 0.12127947807312012 time for calcul the mask position with numpy : 0.02020549774169922 nb_pixel_total : 60987 time to create 1 rle with old method : 0.0627436637878418 time for calcul the mask position with numpy : 0.019376277923583984 nb_pixel_total : 57524 time to create 1 rle with old method : 0.05917692184448242 time for calcul the mask position with numpy : 0.019445419311523438 nb_pixel_total : 10607 time to create 1 rle with old method : 0.011187314987182617 time for calcul the mask position with numpy : 0.01921868324279785 nb_pixel_total : 12768 time to create 1 rle with old method : 0.013365745544433594 time for calcul the mask position with numpy : 0.019845247268676758 nb_pixel_total : 72357 time to create 1 rle with old method : 0.09058141708374023 time for calcul the mask position with numpy : 0.022403478622436523 nb_pixel_total : 69019 time to create 1 rle with old method : 0.07385921478271484 time for calcul the mask position with numpy : 0.020502090454101562 nb_pixel_total : 73934 time to create 1 rle with old method : 0.07781219482421875 time for calcul the mask position with numpy : 0.019727230072021484 nb_pixel_total : 6425 time to create 1 rle with old method : 0.006677150726318359 time for calcul the mask position with numpy : 0.01927495002746582 nb_pixel_total : 1083 time to create 1 rle with old method : 0.0011603832244873047 time for calcul the mask position with numpy : 0.020201444625854492 nb_pixel_total : 11404 time to create 1 rle with old method : 0.012320280075073242 time for calcul the mask position with numpy : 0.021625041961669922 nb_pixel_total : 23014 time to create 1 rle with old method : 0.024626970291137695 time for calcul the mask position with numpy : 0.021582841873168945 nb_pixel_total : 12041 time to create 1 rle with old method : 0.013319253921508789 create new chi : 2.0583817958831787 time to delete rle : 0.0009455680847167969 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 10833 TO DO : save crop sub photo not yet done ! save time : 0.6468899250030518 nb_obj : 18 nb_hashtags : 4 time to prepare the origin masks : 5.170274972915649 time for calcul the mask position with numpy : 0.31304121017456055 nb_pixel_total : 6484764 time to create 1 rle with new method : 0.8839149475097656 time for calcul the mask position with numpy : 0.020255327224731445 nb_pixel_total : 1203 time to create 1 rle with old method : 0.0012941360473632812 time for calcul the mask position with numpy : 0.019527196884155273 nb_pixel_total : 2892 time to create 1 rle with old method : 0.0031616687774658203 time for calcul the mask position with numpy : 0.019615888595581055 nb_pixel_total : 21307 time to create 1 rle with old method : 0.022157669067382812 time for calcul the mask position with numpy : 0.019748210906982422 nb_pixel_total : 757 time to create 1 rle with old method : 0.0008769035339355469 time for calcul the mask position with numpy : 0.0207061767578125 nb_pixel_total : 287736 time to create 1 rle with new method : 0.2339951992034912 time for calcul the mask position with numpy : 0.020791053771972656 nb_pixel_total : 8642 time to create 1 rle with old method : 0.00905919075012207 time for calcul the mask position with numpy : 0.020659685134887695 nb_pixel_total : 18284 time to create 1 rle with old method : 0.019037961959838867 time for calcul the mask position with numpy : 0.019858837127685547 nb_pixel_total : 15208 time to create 1 rle with old method : 0.016047000885009766 time for calcul the mask position with numpy : 0.019719839096069336 nb_pixel_total : 7454 time to create 1 rle with old method : 0.007898807525634766 time for calcul the mask position with numpy : 0.019939422607421875 nb_pixel_total : 1449 time to create 1 rle with old method : 0.0016164779663085938 time for calcul the mask position with numpy : 0.02000737190246582 nb_pixel_total : 25302 time to create 1 rle with old method : 0.02637338638305664 time for calcul the mask position with numpy : 0.019763708114624023 nb_pixel_total : 3690 time to create 1 rle with old method : 0.00406956672668457 time for calcul the mask position with numpy : 0.01988959312438965 nb_pixel_total : 13274 time to create 1 rle with old method : 0.014327287673950195 time for calcul the mask position with numpy : 0.0200653076171875 nb_pixel_total : 2032 time to create 1 rle with old method : 0.0022335052490234375 time for calcul the mask position with numpy : 0.02090740203857422 nb_pixel_total : 4244 time to create 1 rle with old method : 0.0044536590576171875 time for calcul the mask position with numpy : 0.02060985565185547 nb_pixel_total : 94252 time to create 1 rle with old method : 0.09786128997802734 time for calcul the mask position with numpy : 0.021060466766357422 nb_pixel_total : 7278 time to create 1 rle with old method : 0.007620096206665039 time for calcul the mask position with numpy : 0.020603179931640625 nb_pixel_total : 50472 time to create 1 rle with old method : 0.0509943962097168 create new chi : 2.1355679035186768 time to delete rle : 0.0007772445678710938 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 8154 TO DO : save crop sub photo not yet done ! save time : 0.5567483901977539 nb_obj : 8 nb_hashtags : 5 time to prepare the origin masks : 2.468273401260376 time for calcul the mask position with numpy : 0.0506587028503418 nb_pixel_total : 2841846 time to create 1 rle with new method : 0.14258742332458496 time for calcul the mask position with numpy : 0.021358489990234375 nb_pixel_total : 99440 time to create 1 rle with old method : 0.10483288764953613 time for calcul the mask position with numpy : 0.020849943161010742 nb_pixel_total : 52960 time to create 1 rle with old method : 0.0562741756439209 time for calcul the mask position with numpy : 0.020088911056518555 nb_pixel_total : 38554 time to create 1 rle with old method : 0.040750980377197266 time for calcul the mask position with numpy : 0.020159006118774414 nb_pixel_total : 47847 time to create 1 rle with old method : 0.050321102142333984 time for calcul the mask position with numpy : 0.049199819564819336 nb_pixel_total : 3697169 time to create 1 rle with new method : 0.13364124298095703 time for calcul the mask position with numpy : 0.018975019454956055 nb_pixel_total : 4452 time to create 1 rle with old method : 0.0046503543853759766 time for calcul the mask position with numpy : 0.018812179565429688 nb_pixel_total : 5091 time to create 1 rle with old method : 0.0052449703216552734 time for calcul the mask position with numpy : 0.021292448043823242 nb_pixel_total : 262881 time to create 1 rle with new method : 0.12762141227722168 create new chi : 0.9514739513397217 time to delete rle : 0.0008740425109863281 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 10363 TO DO : save crop sub photo not yet done ! save time : 1.076869010925293 nb_obj : 21 nb_hashtags : 7 time to prepare the origin masks : 8.947128295898438 time for calcul the mask position with numpy : 0.14940643310546875 nb_pixel_total : 5047614 time to create 1 rle with new method : 0.739356517791748 time for calcul the mask position with numpy : 0.021181344985961914 nb_pixel_total : 87685 time to create 1 rle with old method : 0.09256291389465332 time for calcul the mask position with numpy : 0.020262479782104492 nb_pixel_total : 1247 time to create 1 rle with old method : 0.0014748573303222656 time for calcul the mask position with numpy : 0.02033400535583496 nb_pixel_total : 38454 time to create 1 rle with old method : 0.04044461250305176 time for calcul the mask position with numpy : 0.020210981369018555 nb_pixel_total : 15850 time to create 1 rle with old method : 0.016933917999267578 time for calcul the mask position with numpy : 0.026284217834472656 nb_pixel_total : 891413 time to create 1 rle with new method : 0.13121676445007324 time for calcul the mask position with numpy : 0.025641441345214844 nb_pixel_total : 192098 time to create 1 rle with new method : 0.13163328170776367 time for calcul the mask position with numpy : 0.02318429946899414 nb_pixel_total : 145284 time to create 1 rle with old method : 0.16930437088012695 time for calcul the mask position with numpy : 0.020980358123779297 nb_pixel_total : 12094 time to create 1 rle with old method : 0.013020992279052734 time for calcul the mask position with numpy : 0.0198209285736084 nb_pixel_total : 457 time to create 1 rle with old method : 0.0010638236999511719 time for calcul the mask position with numpy : 0.01988673210144043 nb_pixel_total : 2821 time to create 1 rle with old method : 0.0031175613403320312 time for calcul the mask position with numpy : 0.019877910614013672 nb_pixel_total : 18031 time to create 1 rle with old method : 0.019176959991455078 time for calcul the mask position with numpy : 0.02187657356262207 nb_pixel_total : 206888 time to create 1 rle with new method : 0.1444385051727295 time for calcul the mask position with numpy : 0.035722970962524414 nb_pixel_total : 88373 time to create 1 rle with old method : 0.09702467918395996 time for calcul the mask position with numpy : 0.04058218002319336 nb_pixel_total : 180870 time to create 1 rle with new method : 0.15151143074035645 time for calcul the mask position with numpy : 0.03480887413024902 nb_pixel_total : 6334 time to create 1 rle with old method : 0.007059574127197266 time for calcul the mask position with numpy : 0.03466343879699707 nb_pixel_total : 17051 time to create 1 rle with old method : 0.017904281616210938 time for calcul the mask position with numpy : 0.03293895721435547 nb_pixel_total : 2971 time to create 1 rle with old method : 0.003119945526123047 time for calcul the mask position with numpy : 0.039283037185668945 nb_pixel_total : 6150 time to create 1 rle with old method : 0.006593942642211914 time for calcul the mask position with numpy : 0.033563852310180664 nb_pixel_total : 29243 time to create 1 rle with old method : 0.04295802116394043 time for calcul the mask position with numpy : 0.03393983840942383 nb_pixel_total : 13736 time to create 1 rle with old method : 0.018651485443115234 time for calcul the mask position with numpy : 0.041449546813964844 nb_pixel_total : 45576 time to create 1 rle with old method : 0.05914950370788574 create new chi : 2.7250826358795166 time to delete rle : 0.0015430450439453125 batch 1 Loaded 22 chid ids of type : 3726 Number RLEs to save : 15335 TO DO : save crop sub photo not yet done ! save time : 0.8320848941802979 map_output_result : {1334704064: (0.0, 'Should be the crop_list due to order', 0.0), 1334704062: (0.0, 'Should be the crop_list due to order', 0.0), 1334704060: (0.0, 'Should be the crop_list due to order', 0.0), 1334703877: (0.0, 'Should be the crop_list due to order', 0.0), 1334703874: (0.0, 'Should be the crop_list due to order', 0.0), 1334703870: (0.0, 'Should be the crop_list due to order', 0.0), 1334703864: (0.0, 'Should be the crop_list due to order', 0.0), 1334703861: (0.0, 'Should be the crop_list due to order', 0.0), 1334703859: (0.0, 'Should be the crop_list due to order', 0.0), 1334703710: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1334704064, 1334704062, 1334704060, 1334703877, 1334703874, 1334703870, 1334703864, 1334703861, 1334703859, 1334703710] Looping around the photos to save general results len do output : 10 /1334704064.Didn't retrieve data . /1334704062.Didn't retrieve data . /1334704060.Didn't retrieve data . /1334703877.Didn't retrieve data . /1334703874.Didn't retrieve data . /1334703870.Didn't retrieve data . /1334703864.Didn't retrieve data . /1334703861.Didn't retrieve data . /1334703859.Didn't retrieve data . /1334703710.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704064', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704062', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704060', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703877', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703874', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703870', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703864', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703861', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703859', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703710', None, None, None, None, None, '2551122') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.0137176513671875 save_final save missing photos in datou_result : time spend for datou_step_exec : 81.92001032829285 time spend to save output : 0.014221429824829102 total time spend for step 5 : 81.93423175811768 step6:crop_condition Wed Feb 5 11:57:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 161 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738753053_3524430 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.0900330543518066 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738753059_3524430 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.967052936553955 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738753064_3524430 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.3758842945098877 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! About to upload 10 photos upload in portfolio : 4869462 init cache_photo without model_param we have 10 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738753069_3524430 we have uploaded 10 photos in the portfolio 4869462 time of upload the photos Elapsed time : 6.8373425006866455 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738753077_3524430 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8629148006439209 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738753079_3524430 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8014154434204102 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1334704064, 1334704062, 1334704060, 1334703877, 1334703874, 1334703870, 1334703864, 1334703861, 1334703859, 1334703710] Looping around the photos to save general results len do output : 28 /1334902885Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334902887Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334902889Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334902891Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334902985Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334902987Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334902989Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334902991Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334902993Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334902995Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903079Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903081Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903083Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903085Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903087Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903154Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903155Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903156Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903157Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903158Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903159Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903160Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903161Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903162Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903167Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903222Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903224Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334903272Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704064', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704062', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704060', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703877', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703874', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703870', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703864', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703861', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703859', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703710', None, None, None, None, None, '2551122') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 94 time used for this insertion : 0.030211687088012695 save_final save missing photos in datou_result : time spend for datou_step_exec : 30.48739743232727 time spend to save output : 0.031264543533325195 total time spend for step 6 : 30.518661975860596 step7:ventilate_hashtags_in_portfolio Wed Feb 5 11:58:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20254737 get user id for portfolio 20254737 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254737 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('autre_refus','environnement','Carton_gris','Teint_Dans_La_Masse','kraft','Carton_brun','flou','mal_croppe','plastique','cartonnette','metal','papier')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254737 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('autre_refus','environnement','Carton_gris','Teint_Dans_La_Masse','kraft','Carton_brun','flou','mal_croppe','plastique','cartonnette','metal','papier')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254737 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('autre_refus','environnement','Carton_gris','Teint_Dans_La_Masse','kraft','Carton_brun','flou','mal_croppe','plastique','cartonnette','metal','papier')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20255628,20255629,20255630,20255631,20255632,20255633,20255634,20255635,20255636,20255637,20255638,20255639?tags=autre_refus,environnement,Carton_gris,Teint_Dans_La_Masse,kraft,Carton_brun,flou,mal_croppe,plastique,cartonnette,metal,papier Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1334704064, 1334704062, 1334704060, 1334703877, 1334703874, 1334703870, 1334703864, 1334703861, 1334703859, 1334703710] Looping around the photos to save general results len do output : 1 /20254737. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704064', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704062', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704060', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703877', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703874', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703870', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703864', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703861', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703859', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703710', None, None, None, None, None, '2551122') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01650524139404297 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.689145088195801 time spend to save output : 0.016798019409179688 total time spend for step 7 : 3.7059431076049805 step8:final Wed Feb 5 11:58:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1334704064: ('0.08109735543377233',), 1334704062: ('0.08109735543377233',), 1334704060: ('0.08109735543377233',), 1334703877: ('0.08109735543377233',), 1334703874: ('0.08109735543377233',), 1334703870: ('0.08109735543377233',), 1334703864: ('0.08109735543377233',), 1334703861: ('0.08109735543377233',), 1334703859: ('0.08109735543377233',), 1334703710: ('0.08109735543377233',)} new output for save of step final : {1334704064: ('0.08109735543377233',), 1334704062: ('0.08109735543377233',), 1334704060: ('0.08109735543377233',), 1334703877: ('0.08109735543377233',), 1334703874: ('0.08109735543377233',), 1334703870: ('0.08109735543377233',), 1334703864: ('0.08109735543377233',), 1334703861: ('0.08109735543377233',), 1334703859: ('0.08109735543377233',), 1334703710: ('0.08109735543377233',)} [1334704064, 1334704062, 1334704060, 1334703877, 1334703874, 1334703870, 1334703864, 1334703861, 1334703859, 1334703710] Looping around the photos to save general results len do output : 10 /1334704064.Didn't retrieve data . /1334704062.Didn't retrieve data . /1334704060.Didn't retrieve data . /1334703877.Didn't retrieve data . /1334703874.Didn't retrieve data . /1334703870.Didn't retrieve data . /1334703864.Didn't retrieve data . /1334703861.Didn't retrieve data . /1334703859.Didn't retrieve data . /1334703710.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704064', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704062', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704060', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703877', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703874', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703870', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703864', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703861', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703859', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703710', None, None, None, None, None, '2551122') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013534069061279297 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.38111019134521484 time spend to save output : 0.014089345932006836 total time spend for step 8 : 0.3951995372772217 step9:velours_tree Wed Feb 5 11:58:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.02992534637451172 time spend to save output : 3.2901763916015625e-05 total time spend for step 9 : 0.029958248138427734 step10:send_mail_cod Wed Feb 5 11:58:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20254737_05-02-2025_11_58_05.pdf 20255628 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202556281738753085 20255632 change filename to text .imagette202556321738753085 20255634 imagette202556341738753085 20255635 imagette202556351738753085 20255636 change filename to text .change filename to text .imagette202556361738753085 20255637 imagette202556371738753085 20255638 imagette202556381738753085 20255630 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202556301738753085 20255631 change filename to text .change filename to text .change filename to text .change filename to text .imagette202556311738753086 20255633 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202556331738753086 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20254737 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20255628,20255629,20255630,20255631,20255632,20255633,20255634,20255635,20255636,20255637,20255638,20255639?tags=autre_refus,environnement,Carton_gris,Teint_Dans_La_Masse,kraft,Carton_brun,flou,mal_croppe,plastique,cartonnette,metal,papier your option no_mail is active, we will not send the real mail to your client args[1334704064] : ('0.08109735543377233',) no score found for photo 1334704064 We are sending mail with results at report@fotonower.com args[1334704062] : ('0.08109735543377233',) no score found for photo 1334704062 We are sending mail with results at report@fotonower.com args[1334704060] : ('0.08109735543377233',) no score found for photo 1334704060 We are sending mail with results at report@fotonower.com args[1334703877] : ('0.08109735543377233',) no score found for photo 1334703877 We are sending mail with results at report@fotonower.com args[1334703874] : ('0.08109735543377233',) no score found for photo 1334703874 We are sending mail with results at report@fotonower.com args[1334703870] : ('0.08109735543377233',) no score found for photo 1334703870 We are sending mail with results at report@fotonower.com args[1334703864] : ('0.08109735543377233',) no score found for photo 1334703864 We are sending mail with results at report@fotonower.com args[1334703861] : ('0.08109735543377233',) no score found for photo 1334703861 We are sending mail with results at report@fotonower.com args[1334703859] : ('0.08109735543377233',) no score found for photo 1334703859 We are sending mail with results at report@fotonower.com args[1334703710] : ('0.08109735543377233',) no score found for photo 1334703710 We are sending mail with results at report@fotonower.com refus_total : 0.08109735543377233 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20254737 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1334596788,1334703710,1334703666,1334703662,1334703660,1334703630,1334703627,1334596919,1334703861,1334596877,1334703870,1334596803,1334596799,1334596796,1334596792,1334596787,1334596643,1334596586,1334596582,1334596579) Found this number of photos: 20 begin to download photo : 1334596788 begin to download photo : 1334703630 begin to download photo : 1334703870 begin to download photo : 1334596787 download finish for photo 1334703630 begin to download photo : 1334703627 download finish for photo 1334596787 begin to download photo : 1334596643 download finish for photo 1334596788 begin to download photo : 1334703710 download finish for photo 1334703870 begin to download photo : 1334596803 download finish for photo 1334703627 begin to download photo : 1334596919 download finish for photo 1334596643 begin to download photo : 1334596586 download finish for photo 1334703710 begin to download photo : 1334703666 download finish for photo 1334596803 begin to download photo : 1334596799 download finish for photo 1334596919 begin to download photo : 1334703861 download finish for photo 1334596586 begin to download photo : 1334596582 download finish for photo 1334703666 begin to download photo : 1334703662 download finish for photo 1334703861 begin to download photo : 1334596877 download finish for photo 1334596799 begin to download photo : 1334596796 download finish for photo 1334596582 begin to download photo : 1334596579 download finish for photo 1334703662 begin to download photo : 1334703660 download finish for photo 1334596877 download finish for photo 1334596796 begin to download photo : 1334596792 download finish for photo 1334703660 download finish for photo 1334596579 download finish for photo 1334596792 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254737_05-02-2025_11_58_05.pdf results_Auto_P20254737_05-02-2025_11_58_05.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254737_05-02-2025_11_58_05.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20254737','results_Auto_P20254737_05-02-2025_11_58_05.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254737_05-02-2025_11_58_05.pdf','pdf','','0.8','0.08109735543377233') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1334704064, 1334704062, 1334704060, 1334703877, 1334703874, 1334703870, 1334703864, 1334703861, 1334703859, 1334703710] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704064', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704062', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704060', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703877', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703874', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703870', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703864', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703861', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703859', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703710', None, None, None, None, None, '2551122') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.013378620147705078 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.946319818496704 time spend to save output : 0.01365041732788086 total time spend for step 10 : 6.959970235824585 step11:split_time_score Wed Feb 5 11:58:11 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('16', 39),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 04022025 20254737 Nombre de photos uploadées : 39 / 23040 (0%) 04022025 20254737 Nombre de photos taguées (types de déchets): 0 / 39 (0%) 04022025 20254737 Nombre de photos taguées (volume) : 0 / 39 (0%) elapsed_time : load_data_split_time_score 2.6226043701171875e-06 elapsed_time : order_list_meta_photo_and_scores 8.344650268554688e-06 ??????????????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0019834041595458984 elapsed_time : insert_dashboard_record_day_entry 0.021861553192138672 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254648 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254650 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254653 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254655 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254657 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254660 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254661 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254663 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254664 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254666 order by id desc limit 1 Qualite : 0.03827887555059262 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20208806_04-02-2025_12_29_58.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20208806 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20208806 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254668 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254669 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254671 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254672 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254674 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254675 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254688 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254690 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254691 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254692 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254693 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254694 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254695 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254696 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254697 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254698 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254699 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254700 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254701 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254716 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254727 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254728 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254729 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254730 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254731 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254732 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254733 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254734 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254735 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254736 order by id desc limit 1 Qualite : 0.08109735543377233 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254737_05-02-2025_11_58_05.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254737 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254737 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254738 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254739 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254740 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254741 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254742 order by id desc limit 1 Qualite : 0.09993071763572346 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254767_05-02-2025_11_50_14.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254767 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254767 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254768 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254769 order by id desc limit 1 Qualite : 0.2029836809337933 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254770_05-02-2025_11_49_33.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254770 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254770 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254771 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254772 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254773 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'04022025': {'nb_upload': 39, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1334704064, 1334704062, 1334704060, 1334703877, 1334703874, 1334703870, 1334703864, 1334703861, 1334703859, 1334703710] Looping around the photos to save general results len do output : 1 /20254737Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704064', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704062', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334704060', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703877', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703874', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703870', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703864', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703861', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703859', None, None, None, None, None, '2551122') ('3459', None, None, None, None, None, None, None, '2551122') ('3459', '20254737', '1334703710', None, None, None, None, None, '2551122') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014001607894897461 save_final save missing photos in datou_result : time spend for datou_step_exec : 39.55470824241638 time spend to save output : 0.014243364334106445 total time spend for step 11 : 39.56895160675049 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 143.64user 66.58system 4:26.54elapsed 78%CPU (0avgtext+0avgdata 4417864maxresident)k 54272inputs+127624outputs (8major+7777211minor)pagefaults 0swaps