python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 72211 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 28>10 please execute split_portfolio.py -i 20461712 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2577764'] with mtr_portfolio_ids : ['20461712'] and first list_photo_ids : [] new path : /proc/72211/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.0870447158813477 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue Feb 11 21:22:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10553 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-11 21:22:35.769089: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-11 21:22:35.807080: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-11 21:22:35.809524: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fbe50000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-11 21:22:35.809585: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-11 21:22:35.821794: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-11 21:22:36.065761: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x37a2d170 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-11 21:22:36.065811: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-11 21:22:36.067728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 21:22:36.070188: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 21:22:36.082404: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 21:22:36.101493: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 21:22:36.104877: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 21:22:36.137972: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 21:22:36.144424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 21:22:36.215953: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 21:22:36.218562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 21:22:36.219403: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 21:22:36.221687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-11 21:22:36.221725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-11 21:22:36.221745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-11 21:22:36.224505: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9777 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-11 21:22:36.668482: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 21:22:36.668594: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 21:22:36.668619: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 21:22:36.668641: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 21:22:36.668663: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 21:22:36.668684: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 21:22:36.668708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 21:22:36.668731: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 21:22:36.670483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 21:22:36.671991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 21:22:36.672050: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 21:22:36.672075: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 21:22:36.672098: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 21:22:36.672121: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 21:22:36.672144: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 21:22:36.672166: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 21:22:36.672188: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 21:22:36.673915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 21:22:36.673951: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-11 21:22:36.673965: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-11 21:22:36.673977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-11 21:22:36.675773: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9777 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-11 21:22:48.780424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 21:22:49.564962: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 33 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 33 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 26 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 10 Detection mask done ! Trying to reset tf kernel 72599 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5264 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10553 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0009539127349853516 nb_pixel_total : 23301 time to create 1 rle with old method : 0.02620840072631836 length of segment : 339 time for calcul the mask position with numpy : 0.002362966537475586 nb_pixel_total : 79389 time to create 1 rle with old method : 0.08742213249206543 length of segment : 325 time for calcul the mask position with numpy : 0.00023865699768066406 nb_pixel_total : 6990 time to create 1 rle with old method : 0.008486270904541016 length of segment : 110 time for calcul the mask position with numpy : 0.0037932395935058594 nb_pixel_total : 120212 time to create 1 rle with old method : 0.13269305229187012 length of segment : 553 time for calcul the mask position with numpy : 0.0003643035888671875 nb_pixel_total : 4940 time to create 1 rle with old method : 0.006345510482788086 length of segment : 56 time for calcul the mask position with numpy : 0.0009038448333740234 nb_pixel_total : 26198 time to create 1 rle with old method : 0.030714988708496094 length of segment : 177 time for calcul the mask position with numpy : 0.0008339881896972656 nb_pixel_total : 25581 time to create 1 rle with old method : 0.02976059913635254 length of segment : 360 time for calcul the mask position with numpy : 0.004657268524169922 nb_pixel_total : 163082 time to create 1 rle with new method : 0.015129566192626953 length of segment : 391 time for calcul the mask position with numpy : 0.0016264915466308594 nb_pixel_total : 47310 time to create 1 rle with old method : 0.052396535873413086 length of segment : 526 time for calcul the mask position with numpy : 0.0004000663757324219 nb_pixel_total : 16794 time to create 1 rle with old method : 0.019086122512817383 length of segment : 201 time for calcul the mask position with numpy : 0.0011763572692871094 nb_pixel_total : 42666 time to create 1 rle with old method : 0.04642033576965332 length of segment : 296 time for calcul the mask position with numpy : 0.0007412433624267578 nb_pixel_total : 37607 time to create 1 rle with old method : 0.040719032287597656 length of segment : 308 time for calcul the mask position with numpy : 0.00020265579223632812 nb_pixel_total : 6012 time to create 1 rle with old method : 0.007491111755371094 length of segment : 142 time for calcul the mask position with numpy : 0.00074005126953125 nb_pixel_total : 26933 time to create 1 rle with old method : 0.03179144859313965 length of segment : 163 time for calcul the mask position with numpy : 0.00160980224609375 nb_pixel_total : 55444 time to create 1 rle with old method : 0.0619349479675293 length of segment : 514 time for calcul the mask position with numpy : 0.0003097057342529297 nb_pixel_total : 10978 time to create 1 rle with old method : 0.012773513793945312 length of segment : 136 time for calcul the mask position with numpy : 0.0011394023895263672 nb_pixel_total : 25429 time to create 1 rle with old method : 0.029036998748779297 length of segment : 528 time for calcul the mask position with numpy : 0.0008442401885986328 nb_pixel_total : 25397 time to create 1 rle with old method : 0.0297088623046875 length of segment : 333 time for calcul the mask position with numpy : 0.00029659271240234375 nb_pixel_total : 8897 time to create 1 rle with old method : 0.010528564453125 length of segment : 143 time for calcul the mask position with numpy : 0.0021843910217285156 nb_pixel_total : 25567 time to create 1 rle with old method : 0.029721975326538086 length of segment : 259 time for calcul the mask position with numpy : 0.0024504661560058594 nb_pixel_total : 103878 time to create 1 rle with old method : 0.12935662269592285 length of segment : 454 time for calcul the mask position with numpy : 0.0009963512420654297 nb_pixel_total : 42585 time to create 1 rle with old method : 0.05229520797729492 length of segment : 288 time for calcul the mask position with numpy : 0.0024785995483398438 nb_pixel_total : 110748 time to create 1 rle with old method : 0.12440013885498047 length of segment : 439 time for calcul the mask position with numpy : 0.0010006427764892578 nb_pixel_total : 29235 time to create 1 rle with old method : 0.030941247940063477 length of segment : 429 time for calcul the mask position with numpy : 0.00092315673828125 nb_pixel_total : 29938 time to create 1 rle with old method : 0.034044504165649414 length of segment : 381 time for calcul the mask position with numpy : 0.0031211376190185547 nb_pixel_total : 115455 time to create 1 rle with old method : 0.13242578506469727 length of segment : 672 time for calcul the mask position with numpy : 0.00044465065002441406 nb_pixel_total : 9825 time to create 1 rle with old method : 0.01154637336730957 length of segment : 202 time for calcul the mask position with numpy : 0.0527803897857666 nb_pixel_total : 2217008 time to create 1 rle with new method : 0.15547990798950195 length of segment : 3468 time for calcul the mask position with numpy : 0.0021305084228515625 nb_pixel_total : 142820 time to create 1 rle with old method : 0.15789222717285156 length of segment : 497 time for calcul the mask position with numpy : 0.0005185604095458984 nb_pixel_total : 13076 time to create 1 rle with old method : 0.016074419021606445 length of segment : 268 time for calcul the mask position with numpy : 0.0007541179656982422 nb_pixel_total : 35915 time to create 1 rle with old method : 0.04151582717895508 length of segment : 325 time for calcul the mask position with numpy : 0.02271270751953125 nb_pixel_total : 877837 time to create 1 rle with new method : 0.05566668510437012 length of segment : 1874 time for calcul the mask position with numpy : 0.0002510547637939453 nb_pixel_total : 11165 time to create 1 rle with old method : 0.014435768127441406 length of segment : 142 time for calcul the mask position with numpy : 0.0007998943328857422 nb_pixel_total : 54192 time to create 1 rle with old method : 0.06381678581237793 length of segment : 253 time for calcul the mask position with numpy : 0.00209808349609375 nb_pixel_total : 126449 time to create 1 rle with old method : 0.14863109588623047 length of segment : 431 time for calcul the mask position with numpy : 0.007800102233886719 nb_pixel_total : 381249 time to create 1 rle with new method : 0.959414005279541 length of segment : 1498 time for calcul the mask position with numpy : 0.0007166862487792969 nb_pixel_total : 12441 time to create 1 rle with old method : 0.018060684204101562 length of segment : 129 time for calcul the mask position with numpy : 0.006212472915649414 nb_pixel_total : 181160 time to create 1 rle with new method : 0.009261608123779297 length of segment : 531 time for calcul the mask position with numpy : 0.0005502700805664062 nb_pixel_total : 4191 time to create 1 rle with old method : 0.006407022476196289 length of segment : 119 time for calcul the mask position with numpy : 0.004130840301513672 nb_pixel_total : 122026 time to create 1 rle with old method : 0.1503920555114746 length of segment : 407 time for calcul the mask position with numpy : 0.000385284423828125 nb_pixel_total : 11179 time to create 1 rle with old method : 0.013177633285522461 length of segment : 201 time for calcul the mask position with numpy : 0.0005052089691162109 nb_pixel_total : 5567 time to create 1 rle with old method : 0.007384300231933594 length of segment : 356 time for calcul the mask position with numpy : 0.0013854503631591797 nb_pixel_total : 49485 time to create 1 rle with old method : 0.059356689453125 length of segment : 400 time for calcul the mask position with numpy : 0.0009167194366455078 nb_pixel_total : 19550 time to create 1 rle with old method : 0.02423405647277832 length of segment : 239 time for calcul the mask position with numpy : 0.0016090869903564453 nb_pixel_total : 42770 time to create 1 rle with old method : 0.049080610275268555 length of segment : 276 time for calcul the mask position with numpy : 0.0002560615539550781 nb_pixel_total : 4364 time to create 1 rle with old method : 0.005591154098510742 length of segment : 85 time for calcul the mask position with numpy : 0.002297639846801758 nb_pixel_total : 47522 time to create 1 rle with old method : 0.0745384693145752 length of segment : 251 time for calcul the mask position with numpy : 0.0004031658172607422 nb_pixel_total : 20418 time to create 1 rle with old method : 0.023512601852416992 length of segment : 200 time for calcul the mask position with numpy : 0.0009517669677734375 nb_pixel_total : 23096 time to create 1 rle with old method : 0.027336835861206055 length of segment : 240 time for calcul the mask position with numpy : 0.00046944618225097656 nb_pixel_total : 8551 time to create 1 rle with old method : 0.010238885879516602 length of segment : 236 time for calcul the mask position with numpy : 0.0004458427429199219 nb_pixel_total : 16903 time to create 1 rle with old method : 0.02028942108154297 length of segment : 205 time for calcul the mask position with numpy : 0.0001392364501953125 nb_pixel_total : 3674 time to create 1 rle with old method : 0.004679203033447266 length of segment : 49 time for calcul the mask position with numpy : 0.0004105567932128906 nb_pixel_total : 14521 time to create 1 rle with old method : 0.017131328582763672 length of segment : 232 time for calcul the mask position with numpy : 0.0012574195861816406 nb_pixel_total : 58305 time to create 1 rle with old method : 0.06667518615722656 length of segment : 310 time for calcul the mask position with numpy : 0.0003204345703125 nb_pixel_total : 9207 time to create 1 rle with old method : 0.01251530647277832 length of segment : 152 time for calcul the mask position with numpy : 0.0008635520935058594 nb_pixel_total : 20874 time to create 1 rle with old method : 0.024158716201782227 length of segment : 279 time for calcul the mask position with numpy : 0.0002601146697998047 nb_pixel_total : 7790 time to create 1 rle with old method : 0.009676218032836914 length of segment : 76 time for calcul the mask position with numpy : 0.0003573894500732422 nb_pixel_total : 10916 time to create 1 rle with old method : 0.013139724731445312 length of segment : 212 time for calcul the mask position with numpy : 0.0012445449829101562 nb_pixel_total : 29730 time to create 1 rle with old method : 0.036176204681396484 length of segment : 230 time for calcul the mask position with numpy : 0.0013010501861572266 nb_pixel_total : 33870 time to create 1 rle with old method : 0.050981760025024414 length of segment : 210 time for calcul the mask position with numpy : 0.0008111000061035156 nb_pixel_total : 32199 time to create 1 rle with old method : 0.03745269775390625 length of segment : 229 time for calcul the mask position with numpy : 0.0009133815765380859 nb_pixel_total : 42309 time to create 1 rle with old method : 0.049043893814086914 length of segment : 245 time for calcul the mask position with numpy : 0.0002770423889160156 nb_pixel_total : 8693 time to create 1 rle with old method : 0.013251781463623047 length of segment : 118 time for calcul the mask position with numpy : 0.002212047576904297 nb_pixel_total : 27380 time to create 1 rle with old method : 0.03323006629943848 length of segment : 395 time for calcul the mask position with numpy : 0.0017197132110595703 nb_pixel_total : 54471 time to create 1 rle with old method : 0.061487674713134766 length of segment : 384 time for calcul the mask position with numpy : 0.0010454654693603516 nb_pixel_total : 43496 time to create 1 rle with old method : 0.04931640625 length of segment : 233 time for calcul the mask position with numpy : 0.01133871078491211 nb_pixel_total : 292827 time to create 1 rle with new method : 0.017040491104125977 length of segment : 741 time for calcul the mask position with numpy : 0.009347200393676758 nb_pixel_total : 205393 time to create 1 rle with new method : 0.014534235000610352 length of segment : 928 time for calcul the mask position with numpy : 0.00030112266540527344 nb_pixel_total : 2885 time to create 1 rle with old method : 0.0034570693969726562 length of segment : 81 time for calcul the mask position with numpy : 0.001226663589477539 nb_pixel_total : 31585 time to create 1 rle with old method : 0.036194801330566406 length of segment : 228 time for calcul the mask position with numpy : 0.0013022422790527344 nb_pixel_total : 29697 time to create 1 rle with old method : 0.03487372398376465 length of segment : 241 time for calcul the mask position with numpy : 0.000850677490234375 nb_pixel_total : 15611 time to create 1 rle with old method : 0.018266916275024414 length of segment : 192 time for calcul the mask position with numpy : 0.001741647720336914 nb_pixel_total : 24471 time to create 1 rle with old method : 0.03133559226989746 length of segment : 254 time for calcul the mask position with numpy : 0.00026988983154296875 nb_pixel_total : 1578 time to create 1 rle with old method : 0.0028159618377685547 length of segment : 62 time for calcul the mask position with numpy : 0.0028243064880371094 nb_pixel_total : 55877 time to create 1 rle with old method : 0.07090258598327637 length of segment : 382 time for calcul the mask position with numpy : 0.006715059280395508 nb_pixel_total : 231639 time to create 1 rle with new method : 0.008899927139282227 length of segment : 593 time for calcul the mask position with numpy : 0.0012350082397460938 nb_pixel_total : 30536 time to create 1 rle with old method : 0.03681588172912598 length of segment : 184 time for calcul the mask position with numpy : 0.0020749568939208984 nb_pixel_total : 66310 time to create 1 rle with old method : 0.07726836204528809 length of segment : 275 time for calcul the mask position with numpy : 0.0020852088928222656 nb_pixel_total : 35471 time to create 1 rle with old method : 0.043016910552978516 length of segment : 276 time for calcul the mask position with numpy : 0.0017092227935791016 nb_pixel_total : 56969 time to create 1 rle with old method : 0.07093167304992676 length of segment : 246 time for calcul the mask position with numpy : 0.0007462501525878906 nb_pixel_total : 14871 time to create 1 rle with old method : 0.017777204513549805 length of segment : 148 time for calcul the mask position with numpy : 0.00032401084899902344 nb_pixel_total : 7083 time to create 1 rle with old method : 0.008557796478271484 length of segment : 65 time for calcul the mask position with numpy : 0.0024139881134033203 nb_pixel_total : 54216 time to create 1 rle with old method : 0.06345748901367188 length of segment : 416 time for calcul the mask position with numpy : 0.0023889541625976562 nb_pixel_total : 75106 time to create 1 rle with old method : 0.08688473701477051 length of segment : 415 time for calcul the mask position with numpy : 0.0012581348419189453 nb_pixel_total : 27323 time to create 1 rle with old method : 0.03192019462585449 length of segment : 163 time for calcul the mask position with numpy : 0.0002815723419189453 nb_pixel_total : 3865 time to create 1 rle with old method : 0.005029916763305664 length of segment : 51 time for calcul the mask position with numpy : 0.004136085510253906 nb_pixel_total : 136191 time to create 1 rle with old method : 0.15590167045593262 length of segment : 434 time for calcul the mask position with numpy : 0.005685329437255859 nb_pixel_total : 153740 time to create 1 rle with new method : 0.012703418731689453 length of segment : 1151 time for calcul the mask position with numpy : 0.0038917064666748047 nb_pixel_total : 66069 time to create 1 rle with old method : 0.08488345146179199 length of segment : 523 time for calcul the mask position with numpy : 0.0022897720336914062 nb_pixel_total : 115858 time to create 1 rle with old method : 0.13048434257507324 length of segment : 510 time for calcul the mask position with numpy : 0.0005862712860107422 nb_pixel_total : 15072 time to create 1 rle with old method : 0.02008843421936035 length of segment : 154 time for calcul the mask position with numpy : 0.0004525184631347656 nb_pixel_total : 8568 time to create 1 rle with old method : 0.012244462966918945 length of segment : 119 time for calcul the mask position with numpy : 0.0016739368438720703 nb_pixel_total : 69795 time to create 1 rle with old method : 0.08104753494262695 length of segment : 292 time for calcul the mask position with numpy : 0.001268625259399414 nb_pixel_total : 32938 time to create 1 rle with old method : 0.04055452346801758 length of segment : 244 time for calcul the mask position with numpy : 0.0012125968933105469 nb_pixel_total : 19901 time to create 1 rle with old method : 0.026028871536254883 length of segment : 304 time for calcul the mask position with numpy : 0.00601649284362793 nb_pixel_total : 247304 time to create 1 rle with new method : 0.011587142944335938 length of segment : 975 time for calcul the mask position with numpy : 0.00018644332885742188 nb_pixel_total : 3042 time to create 1 rle with old method : 0.0038406848907470703 length of segment : 50 time for calcul the mask position with numpy : 0.001749277114868164 nb_pixel_total : 53043 time to create 1 rle with old method : 0.06181621551513672 length of segment : 317 time for calcul the mask position with numpy : 0.000705718994140625 nb_pixel_total : 14819 time to create 1 rle with old method : 0.01887059211730957 length of segment : 139 time for calcul the mask position with numpy : 0.0002067089080810547 nb_pixel_total : 3407 time to create 1 rle with old method : 0.00441741943359375 length of segment : 50 time for calcul the mask position with numpy : 0.003512859344482422 nb_pixel_total : 109042 time to create 1 rle with old method : 0.12490510940551758 length of segment : 648 time for calcul the mask position with numpy : 0.0032281875610351562 nb_pixel_total : 37314 time to create 1 rle with old method : 0.04443216323852539 length of segment : 544 time for calcul the mask position with numpy : 0.006854534149169922 nb_pixel_total : 134610 time to create 1 rle with old method : 0.1790170669555664 length of segment : 904 time for calcul the mask position with numpy : 0.00019550323486328125 nb_pixel_total : 2529 time to create 1 rle with old method : 0.0029783248901367188 length of segment : 72 time for calcul the mask position with numpy : 0.0007612705230712891 nb_pixel_total : 16183 time to create 1 rle with old method : 0.018998146057128906 length of segment : 188 time for calcul the mask position with numpy : 0.002073049545288086 nb_pixel_total : 101595 time to create 1 rle with old method : 0.11984729766845703 length of segment : 792 time for calcul the mask position with numpy : 0.0002384185791015625 nb_pixel_total : 3373 time to create 1 rle with old method : 0.0042264461517333984 length of segment : 68 time for calcul the mask position with numpy : 0.00016927719116210938 nb_pixel_total : 2344 time to create 1 rle with old method : 0.0028519630432128906 length of segment : 66 time for calcul the mask position with numpy : 0.0002875328063964844 nb_pixel_total : 190 time to create 1 rle with old method : 0.0006952285766601562 length of segment : 26 time for calcul the mask position with numpy : 0.0006754398345947266 nb_pixel_total : 35241 time to create 1 rle with old method : 0.04406332969665527 length of segment : 301 time for calcul the mask position with numpy : 0.00022864341735839844 nb_pixel_total : 2690 time to create 1 rle with old method : 0.0036203861236572266 length of segment : 62 time for calcul the mask position with numpy : 0.0006382465362548828 nb_pixel_total : 17305 time to create 1 rle with old method : 0.020521163940429688 length of segment : 165 time for calcul the mask position with numpy : 0.028844118118286133 nb_pixel_total : 1088166 time to create 1 rle with new method : 0.2446610927581787 length of segment : 1458 time for calcul the mask position with numpy : 0.0001442432403564453 nb_pixel_total : 1475 time to create 1 rle with old method : 0.00183868408203125 length of segment : 43 time for calcul the mask position with numpy : 0.0027697086334228516 nb_pixel_total : 120722 time to create 1 rle with old method : 0.13904714584350586 length of segment : 509 time for calcul the mask position with numpy : 0.00015878677368164062 nb_pixel_total : 2579 time to create 1 rle with old method : 0.0034487247467041016 length of segment : 55 time for calcul the mask position with numpy : 0.07228517532348633 nb_pixel_total : 2056639 time to create 1 rle with new method : 0.4009988307952881 length of segment : 2073 time for calcul the mask position with numpy : 0.00023174285888671875 nb_pixel_total : 3957 time to create 1 rle with old method : 0.004908561706542969 length of segment : 76 time for calcul the mask position with numpy : 0.0011861324310302734 nb_pixel_total : 69006 time to create 1 rle with old method : 0.07858037948608398 length of segment : 276 time for calcul the mask position with numpy : 0.005475282669067383 nb_pixel_total : 212684 time to create 1 rle with new method : 0.009629964828491211 length of segment : 748 time for calcul the mask position with numpy : 0.0003190040588378906 nb_pixel_total : 6738 time to create 1 rle with old method : 0.008316516876220703 length of segment : 168 time for calcul the mask position with numpy : 8.869171142578125e-05 nb_pixel_total : 1900 time to create 1 rle with old method : 0.0024445056915283203 length of segment : 71 time for calcul the mask position with numpy : 0.0017247200012207031 nb_pixel_total : 78028 time to create 1 rle with old method : 0.13734221458435059 length of segment : 339 time for calcul the mask position with numpy : 0.0009489059448242188 nb_pixel_total : 38097 time to create 1 rle with old method : 0.044999122619628906 length of segment : 321 time for calcul the mask position with numpy : 0.0014221668243408203 nb_pixel_total : 60576 time to create 1 rle with old method : 0.07281136512756348 length of segment : 252 time for calcul the mask position with numpy : 0.00529026985168457 nb_pixel_total : 338832 time to create 1 rle with new method : 0.009483098983764648 length of segment : 478 time for calcul the mask position with numpy : 0.0008411407470703125 nb_pixel_total : 62517 time to create 1 rle with old method : 0.07338380813598633 length of segment : 352 time for calcul the mask position with numpy : 0.006022930145263672 nb_pixel_total : 220960 time to create 1 rle with new method : 0.009185552597045898 length of segment : 673 time for calcul the mask position with numpy : 0.009621143341064453 nb_pixel_total : 324655 time to create 1 rle with new method : 0.014477968215942383 length of segment : 705 time for calcul the mask position with numpy : 0.03785085678100586 nb_pixel_total : 902486 time to create 1 rle with new method : 0.36907100677490234 length of segment : 2511 time for calcul the mask position with numpy : 0.0011668205261230469 nb_pixel_total : 26433 time to create 1 rle with old method : 0.0306398868560791 length of segment : 405 time for calcul the mask position with numpy : 0.003868579864501953 nb_pixel_total : 268176 time to create 1 rle with new method : 0.0076296329498291016 length of segment : 701 time for calcul the mask position with numpy : 0.00039958953857421875 nb_pixel_total : 12694 time to create 1 rle with old method : 0.014690637588500977 length of segment : 292 time for calcul the mask position with numpy : 0.0016186237335205078 nb_pixel_total : 83693 time to create 1 rle with old method : 0.0946187973022461 length of segment : 874 time for calcul the mask position with numpy : 0.0044209957122802734 nb_pixel_total : 218401 time to create 1 rle with new method : 0.012132883071899414 length of segment : 942 time for calcul the mask position with numpy : 0.00261688232421875 nb_pixel_total : 96554 time to create 1 rle with old method : 0.10736966133117676 length of segment : 509 time for calcul the mask position with numpy : 0.0002353191375732422 nb_pixel_total : 8131 time to create 1 rle with old method : 0.010059118270874023 length of segment : 67 time for calcul the mask position with numpy : 0.0001323223114013672 nb_pixel_total : 4008 time to create 1 rle with old method : 0.0048139095306396484 length of segment : 93 time for calcul the mask position with numpy : 0.02028346061706543 nb_pixel_total : 639660 time to create 1 rle with new method : 0.22306561470031738 length of segment : 1358 time for calcul the mask position with numpy : 0.0007576942443847656 nb_pixel_total : 23772 time to create 1 rle with old method : 0.02745223045349121 length of segment : 267 time for calcul the mask position with numpy : 0.0112762451171875 nb_pixel_total : 528335 time to create 1 rle with new method : 0.021953344345092773 length of segment : 888 time for calcul the mask position with numpy : 0.0034055709838867188 nb_pixel_total : 228875 time to create 1 rle with new method : 0.009018182754516602 length of segment : 664 time for calcul the mask position with numpy : 0.0009949207305908203 nb_pixel_total : 38314 time to create 1 rle with old method : 0.04386639595031738 length of segment : 544 time for calcul the mask position with numpy : 0.0005023479461669922 nb_pixel_total : 22180 time to create 1 rle with old method : 0.02579021453857422 length of segment : 200 time for calcul the mask position with numpy : 0.0006239414215087891 nb_pixel_total : 32382 time to create 1 rle with old method : 0.0371251106262207 length of segment : 294 time for calcul the mask position with numpy : 0.0039327144622802734 nb_pixel_total : 234201 time to create 1 rle with new method : 0.011374950408935547 length of segment : 691 time for calcul the mask position with numpy : 0.00020623207092285156 nb_pixel_total : 8657 time to create 1 rle with old method : 0.010534524917602539 length of segment : 133 time for calcul the mask position with numpy : 0.01063990592956543 nb_pixel_total : 692018 time to create 1 rle with new method : 0.02516031265258789 length of segment : 1015 time for calcul the mask position with numpy : 0.003119945526123047 nb_pixel_total : 128377 time to create 1 rle with old method : 0.14521408081054688 length of segment : 516 time for calcul the mask position with numpy : 0.0001804828643798828 nb_pixel_total : 2413 time to create 1 rle with old method : 0.0029878616333007812 length of segment : 62 time for calcul the mask position with numpy : 0.00013184547424316406 nb_pixel_total : 1559 time to create 1 rle with old method : 0.0019249916076660156 length of segment : 47 time for calcul the mask position with numpy : 0.0006153583526611328 nb_pixel_total : 13380 time to create 1 rle with old method : 0.016289234161376953 length of segment : 152 time for calcul the mask position with numpy : 0.0010669231414794922 nb_pixel_total : 34939 time to create 1 rle with old method : 0.03923439979553223 length of segment : 445 time for calcul the mask position with numpy : 0.0042951107025146484 nb_pixel_total : 204349 time to create 1 rle with new method : 0.008912324905395508 length of segment : 375 time for calcul the mask position with numpy : 0.00022411346435546875 nb_pixel_total : 5470 time to create 1 rle with old method : 0.006576061248779297 length of segment : 71 time for calcul the mask position with numpy : 0.00025343894958496094 nb_pixel_total : 10414 time to create 1 rle with old method : 0.012655973434448242 length of segment : 157 time for calcul the mask position with numpy : 0.00025534629821777344 nb_pixel_total : 3063 time to create 1 rle with old method : 0.004096269607543945 length of segment : 82 time for calcul the mask position with numpy : 0.0008192062377929688 nb_pixel_total : 26299 time to create 1 rle with old method : 0.030649662017822266 length of segment : 354 time for calcul the mask position with numpy : 0.0007975101470947266 nb_pixel_total : 28433 time to create 1 rle with old method : 0.03273606300354004 length of segment : 232 time for calcul the mask position with numpy : 0.0020983219146728516 nb_pixel_total : 59129 time to create 1 rle with old method : 0.06957125663757324 length of segment : 410 time for calcul the mask position with numpy : 0.0007610321044921875 nb_pixel_total : 17192 time to create 1 rle with old method : 0.01960468292236328 length of segment : 283 time for calcul the mask position with numpy : 0.008188962936401367 nb_pixel_total : 314127 time to create 1 rle with new method : 0.015033245086669922 length of segment : 657 time for calcul the mask position with numpy : 0.0017430782318115234 nb_pixel_total : 22903 time to create 1 rle with old method : 0.026668787002563477 length of segment : 422 time for calcul the mask position with numpy : 0.0004706382751464844 nb_pixel_total : 14416 time to create 1 rle with old method : 0.01713705062866211 length of segment : 196 time for calcul the mask position with numpy : 0.00036525726318359375 nb_pixel_total : 7377 time to create 1 rle with old method : 0.008955717086791992 length of segment : 126 time for calcul the mask position with numpy : 0.0007817745208740234 nb_pixel_total : 29333 time to create 1 rle with old method : 0.033380985260009766 length of segment : 186 time for calcul the mask position with numpy : 0.000347137451171875 nb_pixel_total : 4530 time to create 1 rle with old method : 0.00549006462097168 length of segment : 147 time for calcul the mask position with numpy : 0.0005662441253662109 nb_pixel_total : 14115 time to create 1 rle with old method : 0.01685619354248047 length of segment : 213 time for calcul the mask position with numpy : 0.00013709068298339844 nb_pixel_total : 3072 time to create 1 rle with old method : 0.0038361549377441406 length of segment : 38 time for calcul the mask position with numpy : 0.0005266666412353516 nb_pixel_total : 10963 time to create 1 rle with old method : 0.013275623321533203 length of segment : 140 time for calcul the mask position with numpy : 0.00032830238342285156 nb_pixel_total : 16344 time to create 1 rle with old method : 0.019733667373657227 length of segment : 245 time for calcul the mask position with numpy : 0.0001430511474609375 nb_pixel_total : 4541 time to create 1 rle with old method : 0.005842447280883789 length of segment : 99 time for calcul the mask position with numpy : 0.0014109611511230469 nb_pixel_total : 68971 time to create 1 rle with old method : 0.07725214958190918 length of segment : 269 time for calcul the mask position with numpy : 0.032636404037475586 nb_pixel_total : 995973 time to create 1 rle with new method : 0.2907116413116455 length of segment : 1639 time for calcul the mask position with numpy : 0.0002994537353515625 nb_pixel_total : 4918 time to create 1 rle with old method : 0.005794525146484375 length of segment : 145 time for calcul the mask position with numpy : 0.002827882766723633 nb_pixel_total : 185404 time to create 1 rle with new method : 0.005833148956298828 length of segment : 417 time for calcul the mask position with numpy : 0.0009751319885253906 nb_pixel_total : 42079 time to create 1 rle with old method : 0.0516362190246582 length of segment : 188 time for calcul the mask position with numpy : 0.01417994499206543 nb_pixel_total : 818105 time to create 1 rle with new method : 0.051894187927246094 length of segment : 1056 time for calcul the mask position with numpy : 0.00616455078125 nb_pixel_total : 308754 time to create 1 rle with new method : 0.013061285018920898 length of segment : 834 time for calcul the mask position with numpy : 0.0035805702209472656 nb_pixel_total : 280979 time to create 1 rle with new method : 0.01007080078125 length of segment : 782 time spent for convertir_results : 21.41808032989502 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 180 chid ids of type : 3663 Number RLEs to save : 71779 save missing photos in datou_result : time spend for datou_step_exec : 76.35861468315125 time spend to save output : 4.794574737548828 total time spend for step 1 : 81.15318942070007 step2:crop_condition Tue Feb 11 21:23:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 180 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 18 About to insert : list_path_to_insert length 18 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 17 About to insert : list_path_to_insert length 17 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 54 About to insert : list_path_to_insert length 54 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1336884922, 1336884917, 1336867305, 1336867303, 1336867300, 1336867297, 1336867290, 1336867257, 1336867251, 1336867238] Looping around the photos to save general results len do output : 114 /-3670633728Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633727Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633725Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633765Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633773Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633756Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633769Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633790Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633791Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633792Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633811Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633822Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633813Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633845Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633841Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633874Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633886Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633888Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633715Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633735Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633722Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633753Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633750Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633747Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633772Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633783Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633798Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633823Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633816Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633837Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633832Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633858Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633853Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633892Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633890Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633737Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633736Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633738Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633785Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633806Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633803Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633840Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633859Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633866Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633869Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633884Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633721Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633733Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633734Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633729Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633730Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633717Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633724Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633740Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633739Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633731Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633748Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633751Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633742Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633762Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633775Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633758Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633779Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633778Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633760Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633776Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633784Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633809Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633802Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633797Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633796Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633794Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633801Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633807Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633786Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633814Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633815Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633819Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633818Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633838Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633839Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633833Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633828Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633848Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633847Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633842Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633855Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633857Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633849Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633856Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633844Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633846Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633871Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633862Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633867Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633879Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633881Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633873Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633889Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633893Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633726Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633720Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633723Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633770Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633764Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633761Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633788Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633865Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633878Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633754Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633744Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633789Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633800Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670633826Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884922', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884917', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867305', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867303', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867300', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867297', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867290', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867257', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867251', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867238', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 352 time used for this insertion : 0.0445094108581543 save_final save missing photos in datou_result : time spend for datou_step_exec : 49.12389087677002 time spend to save output : 0.047798871994018555 total time spend for step 2 : 49.17168974876404 step3:thcl Tue Feb 11 21:24:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.008485794067382812 time to convert the images to numpy array : 0.09761953353881836 time to import caffe and check if the image exist : 0.016389846801757812 time to convert the images to numpy array : 0.6308214664459229 time to import caffe and check if the image exist : 0.01535177230834961 time to convert the images to numpy array : 0.6609430313110352 time to import caffe and check if the image exist : 0.01713418960571289 time to convert the images to numpy array : 0.6673619747161865 time to import caffe and check if the image exist : 0.011099100112915039 time to convert the images to numpy array : 0.6736977100372314 time to import caffe and check if the image exist : 0.014564752578735352 time to convert the images to numpy array : 0.6716043949127197 time to import caffe and check if the image exist : 0.0187075138092041 time to convert the images to numpy array : 0.7553391456604004 time to import caffe and check if the image exist : 0.01237034797668457 time to convert the images to numpy array : 0.7613227367401123 time to import caffe and check if the image exist : 0.01689887046813965 time to convert the images to numpy array : 0.7853579521179199 time to import caffe and check if the image exist : 0.009889602661132812 time to convert the images to numpy array : 1.1407504081726074 total time to convert the images to numpy array : 1.3599202632904053 list photo_ids error: [] list photo_ids correct : [-3670633728, -3670633727, -3670633725, -3670633765, -3670633773, -3670633756, -3670633769, -3670633790, -3670633791, -3670633792, -3670633811, -3670633822, -3670633881, -3670633873, -3670633889, -3670633893, -3670633726, -3670633720, -3670633723, -3670633770, -3670633764, -3670633761, -3670633788, -3670633865, -3670633775, -3670633758, -3670633779, -3670633778, -3670633760, -3670633776, -3670633784, -3670633809, -3670633802, -3670633797, -3670633796, -3670633794, -3670633813, -3670633845, -3670633841, -3670633874, -3670633886, -3670633888, -3670633715, -3670633735, -3670633722, -3670633753, -3670633750, -3670633747, -3670633736, -3670633738, -3670633785, -3670633806, -3670633803, -3670633840, -3670633859, -3670633866, -3670633869, -3670633884, -3670633721, -3670633733, -3670633734, -3670633729, -3670633730, -3670633717, -3670633724, -3670633740, -3670633739, -3670633731, -3670633748, -3670633751, -3670633742, -3670633762, -3670633772, -3670633783, -3670633798, -3670633823, -3670633816, -3670633837, -3670633832, -3670633858, -3670633853, -3670633892, -3670633890, -3670633737, -3670633847, -3670633842, -3670633855, -3670633857, -3670633849, -3670633856, -3670633844, -3670633846, -3670633871, -3670633862, -3670633867, -3670633879, -3670633801, -3670633807, -3670633786, -3670633814, -3670633815, -3670633819, -3670633818, -3670633838, -3670633839, -3670633833, -3670633828, -3670633848, -3670633878, -3670633754, -3670633744, -3670633789, -3670633800, -3670633826] number of photos to traite : 114 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 9754 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 9141 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.5113294124603271 time used to do the prediction : 0.21592330932617188 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5043 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5043 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.814450979232788 time used to do the prediction : 0.20971155166625977 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.193450927734375e-05 save missing photos in datou_result : time spend for datou_step_exec : 12.208824872970581 time spend to save output : 0.0009160041809082031 total time spend for step 3 : 12.20974087715149 step4:merge_mask_thcl_custom Tue Feb 11 21:24:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 180 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1336884922, 1336884917, 1336867305, 1336867303, 1336867300, 1336867297, 1336867290, 1336867257, 1336867251, 1336867238] Looping around the photos to save general results len do output : 10 /1336884922Didn't retrieve data .Didn't retrieve data . /1336884917Didn't retrieve data .Didn't retrieve data . /1336867305Didn't retrieve data .Didn't retrieve data . /1336867303Didn't retrieve data .Didn't retrieve data . /1336867300Didn't retrieve data .Didn't retrieve data . /1336867297Didn't retrieve data .Didn't retrieve data . /1336867290Didn't retrieve data .Didn't retrieve data . /1336867257Didn't retrieve data .Didn't retrieve data . /1336867251Didn't retrieve data .Didn't retrieve data . /1336867238Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884922', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884917', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867305', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867303', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867300', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867297', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867290', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867257', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867251', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867238', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.016634225845336914 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.06496214866638184 time spend to save output : 0.017147302627563477 total time spend for step 4 : 0.08210945129394531 step5:rle_unique_nms_with_priority Tue Feb 11 21:24:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 26 nb_hashtags : 5 time to prepare the origin masks : 11.931556701660156 time for calcul the mask position with numpy : 0.24552273750305176 nb_pixel_total : 6039860 time to create 1 rle with new method : 0.716059684753418 time for calcul the mask position with numpy : 0.02879619598388672 nb_pixel_total : 23263 time to create 1 rle with old method : 0.027086496353149414 time for calcul the mask position with numpy : 0.028849363327026367 nb_pixel_total : 9750 time to create 1 rle with old method : 0.011370420455932617 time for calcul the mask position with numpy : 0.02870798110961914 nb_pixel_total : 25319 time to create 1 rle with old method : 0.02976703643798828 time for calcul the mask position with numpy : 0.028392791748046875 nb_pixel_total : 6940 time to create 1 rle with old method : 0.00774836540222168 time for calcul the mask position with numpy : 0.02774786949157715 nb_pixel_total : 29159 time to create 1 rle with old method : 0.03262066841125488 time for calcul the mask position with numpy : 0.0286409854888916 nb_pixel_total : 5979 time to create 1 rle with old method : 0.007063150405883789 time for calcul the mask position with numpy : 0.03306913375854492 nb_pixel_total : 47267 time to create 1 rle with old method : 0.0569918155670166 time for calcul the mask position with numpy : 0.029871702194213867 nb_pixel_total : 106367 time to create 1 rle with old method : 0.11839509010314941 time for calcul the mask position with numpy : 0.028900623321533203 nb_pixel_total : 8202 time to create 1 rle with old method : 0.009811878204345703 time for calcul the mask position with numpy : 0.029469013214111328 nb_pixel_total : 162750 time to create 1 rle with new method : 0.9229192733764648 time for calcul the mask position with numpy : 0.028951406478881836 nb_pixel_total : 25540 time to create 1 rle with old method : 0.027816295623779297 time for calcul the mask position with numpy : 0.02885746955871582 nb_pixel_total : 119799 time to create 1 rle with old method : 0.13805651664733887 time for calcul the mask position with numpy : 0.028458833694458008 nb_pixel_total : 10676 time to create 1 rle with old method : 0.011906147003173828 time for calcul the mask position with numpy : 0.02873992919921875 nb_pixel_total : 16753 time to create 1 rle with old method : 0.018422603607177734 time for calcul the mask position with numpy : 0.02874135971069336 nb_pixel_total : 1663 time to create 1 rle with old method : 0.0021097660064697266 time for calcul the mask position with numpy : 0.0291292667388916 nb_pixel_total : 27715 time to create 1 rle with old method : 0.03058934211730957 time for calcul the mask position with numpy : 0.02845311164855957 nb_pixel_total : 1894 time to create 1 rle with old method : 0.0023369789123535156 time for calcul the mask position with numpy : 0.029080629348754883 nb_pixel_total : 79375 time to create 1 rle with old method : 0.08850884437561035 time for calcul the mask position with numpy : 0.03589510917663574 nb_pixel_total : 114870 time to create 1 rle with old method : 0.1287219524383545 time for calcul the mask position with numpy : 0.029643774032592773 nb_pixel_total : 49612 time to create 1 rle with old method : 0.055551767349243164 time for calcul the mask position with numpy : 0.031548500061035156 nb_pixel_total : 1301 time to create 1 rle with old method : 0.0017039775848388672 time for calcul the mask position with numpy : 0.038791656494140625 nb_pixel_total : 36235 time to create 1 rle with old method : 0.05276608467102051 time for calcul the mask position with numpy : 0.043493032455444336 nb_pixel_total : 42520 time to create 1 rle with old method : 0.07117724418640137 time for calcul the mask position with numpy : 0.04140210151672363 nb_pixel_total : 26001 time to create 1 rle with old method : 0.035202980041503906 time for calcul the mask position with numpy : 0.029287338256835938 nb_pixel_total : 26571 time to create 1 rle with old method : 0.031656503677368164 time for calcul the mask position with numpy : 0.02909684181213379 nb_pixel_total : 4859 time to create 1 rle with old method : 0.005859851837158203 create new chi : 3.749546527862549 time to delete rle : 0.015158414840698242 batch 1 Loaded 27 chid ids of type : 3726 Number RLEs to save : 16972 TO DO : save crop sub photo not yet done ! save time : 1.114917278289795 nb_obj : 15 nb_hashtags : 5 time to prepare the origin masks : 10.903741598129272 time for calcul the mask position with numpy : 0.30272769927978516 nb_pixel_total : 3448123 time to create 1 rle with new method : 0.46129417419433594 time for calcul the mask position with numpy : 0.02598118782043457 nb_pixel_total : 9758 time to create 1 rle with old method : 0.011097431182861328 time for calcul the mask position with numpy : 0.026357650756835938 nb_pixel_total : 1739 time to create 1 rle with old method : 0.0031647682189941406 time for calcul the mask position with numpy : 0.026454687118530273 nb_pixel_total : 35787 time to create 1 rle with old method : 0.05460643768310547 time for calcul the mask position with numpy : 0.034615278244018555 nb_pixel_total : 54085 time to create 1 rle with old method : 0.07690620422363281 time for calcul the mask position with numpy : 0.036533355712890625 nb_pixel_total : 119551 time to create 1 rle with old method : 0.13636016845703125 time for calcul the mask position with numpy : 0.03411746025085449 nb_pixel_total : 3634 time to create 1 rle with old method : 0.00604701042175293 time for calcul the mask position with numpy : 0.03119039535522461 nb_pixel_total : 135286 time to create 1 rle with old method : 0.17023444175720215 time for calcul the mask position with numpy : 0.034288644790649414 nb_pixel_total : 4153 time to create 1 rle with old method : 0.004993915557861328 time for calcul the mask position with numpy : 0.04840350151062012 nb_pixel_total : 3 time to create 1 rle with old method : 4.172325134277344e-05 time for calcul the mask position with numpy : 0.056062936782836914 nb_pixel_total : 358768 time to create 1 rle with new method : 0.7978990077972412 time for calcul the mask position with numpy : 0.043949127197265625 nb_pixel_total : 11104 time to create 1 rle with old method : 0.012364387512207031 time for calcul the mask position with numpy : 0.04779052734375 nb_pixel_total : 460791 time to create 1 rle with new method : 1.0817594528198242 time for calcul the mask position with numpy : 0.03663301467895508 nb_pixel_total : 13027 time to create 1 rle with old method : 0.016156673431396484 time for calcul the mask position with numpy : 0.04937863349914551 nb_pixel_total : 2213624 time to create 1 rle with new method : 0.5206851959228516 time for calcul the mask position with numpy : 0.030612468719482422 nb_pixel_total : 180807 time to create 1 rle with new method : 0.6504127979278564 create new chi : 4.998512506484985 time to delete rle : 0.002332925796508789 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 20454 TO DO : save crop sub photo not yet done ! save time : 2.1996569633483887 nb_obj : 29 nb_hashtags : 7 time to prepare the origin masks : 9.17417860031128 time for calcul the mask position with numpy : 1.3188610076904297 nb_pixel_total : 6261342 time to create 1 rle with new method : 0.5441641807556152 time for calcul the mask position with numpy : 0.02969646453857422 nb_pixel_total : 58163 time to create 1 rle with old method : 0.0688929557800293 time for calcul the mask position with numpy : 0.02948617935180664 nb_pixel_total : 20300 time to create 1 rle with old method : 0.02311849594116211 time for calcul the mask position with numpy : 0.02913641929626465 nb_pixel_total : 9189 time to create 1 rle with old method : 0.010830163955688477 time for calcul the mask position with numpy : 0.03264474868774414 nb_pixel_total : 22929 time to create 1 rle with old method : 0.025921344757080078 time for calcul the mask position with numpy : 0.03011322021484375 nb_pixel_total : 194828 time to create 1 rle with new method : 0.6274881362915039 time for calcul the mask position with numpy : 0.029245615005493164 nb_pixel_total : 8487 time to create 1 rle with old method : 0.010178327560424805 time for calcul the mask position with numpy : 0.029211997985839844 nb_pixel_total : 49380 time to create 1 rle with old method : 0.058260440826416016 time for calcul the mask position with numpy : 0.029346704483032227 nb_pixel_total : 32082 time to create 1 rle with old method : 0.037612199783325195 time for calcul the mask position with numpy : 0.02931356430053711 nb_pixel_total : 622 time to create 1 rle with old method : 0.0009407997131347656 time for calcul the mask position with numpy : 0.029182910919189453 nb_pixel_total : 3642 time to create 1 rle with old method : 0.004353046417236328 time for calcul the mask position with numpy : 0.029391050338745117 nb_pixel_total : 4336 time to create 1 rle with old method : 0.005141735076904297 time for calcul the mask position with numpy : 0.029443740844726562 nb_pixel_total : 14498 time to create 1 rle with old method : 0.016488075256347656 time for calcul the mask position with numpy : 0.02930164337158203 nb_pixel_total : 21411 time to create 1 rle with old method : 0.024839162826538086 time for calcul the mask position with numpy : 0.02926182746887207 nb_pixel_total : 10904 time to create 1 rle with old method : 0.012529611587524414 time for calcul the mask position with numpy : 0.029994726181030273 nb_pixel_total : 20773 time to create 1 rle with old method : 0.023067235946655273 time for calcul the mask position with numpy : 0.029057979583740234 nb_pixel_total : 8671 time to create 1 rle with old method : 0.010113716125488281 time for calcul the mask position with numpy : 0.028664588928222656 nb_pixel_total : 42084 time to create 1 rle with old method : 0.04901313781738281 time for calcul the mask position with numpy : 0.02914261817932129 nb_pixel_total : 29536 time to create 1 rle with old method : 0.03307151794433594 time for calcul the mask position with numpy : 0.028948307037353516 nb_pixel_total : 31 time to create 1 rle with old method : 0.0001862049102783203 time for calcul the mask position with numpy : 0.02863168716430664 nb_pixel_total : 47415 time to create 1 rle with old method : 0.05496788024902344 time for calcul the mask position with numpy : 0.029276609420776367 nb_pixel_total : 1261 time to create 1 rle with old method : 0.0017943382263183594 time for calcul the mask position with numpy : 0.02922964096069336 nb_pixel_total : 16833 time to create 1 rle with old method : 0.019466876983642578 time for calcul the mask position with numpy : 0.029232501983642578 nb_pixel_total : 1005 time to create 1 rle with old method : 0.0013458728790283203 time for calcul the mask position with numpy : 0.03291177749633789 nb_pixel_total : 7754 time to create 1 rle with old method : 0.009200811386108398 time for calcul the mask position with numpy : 0.02942943572998047 nb_pixel_total : 42649 time to create 1 rle with old method : 0.04905843734741211 time for calcul the mask position with numpy : 0.03124260902404785 nb_pixel_total : 43321 time to create 1 rle with old method : 0.04897737503051758 time for calcul the mask position with numpy : 0.031992197036743164 nb_pixel_total : 54393 time to create 1 rle with old method : 0.0621945858001709 time for calcul the mask position with numpy : 0.029378175735473633 nb_pixel_total : 19519 time to create 1 rle with old method : 0.023032426834106445 time for calcul the mask position with numpy : 0.03254127502441406 nb_pixel_total : 2882 time to create 1 rle with old method : 0.003545522689819336 create new chi : 4.098031520843506 time to delete rle : 0.0015037059783935547 batch 1 Loaded 30 chid ids of type : 3726 Number RLEs to save : 14982 TO DO : save crop sub photo not yet done ! save time : 0.8832061290740967 nb_obj : 23 nb_hashtags : 7 time to prepare the origin masks : 21.874127864837646 time for calcul the mask position with numpy : 0.38384509086608887 nb_pixel_total : 5724735 time to create 1 rle with new method : 0.936854362487793 time for calcul the mask position with numpy : 0.04639792442321777 nb_pixel_total : 1087 time to create 1 rle with old method : 0.0018973350524902344 time for calcul the mask position with numpy : 0.04520726203918457 nb_pixel_total : 26989 time to create 1 rle with old method : 0.03029036521911621 time for calcul the mask position with numpy : 0.045018911361694336 nb_pixel_total : 69696 time to create 1 rle with old method : 0.07871055603027344 time for calcul the mask position with numpy : 0.04823732376098633 nb_pixel_total : 19718 time to create 1 rle with old method : 0.023281574249267578 time for calcul the mask position with numpy : 0.044799089431762695 nb_pixel_total : 24398 time to create 1 rle with old method : 0.02903580665588379 time for calcul the mask position with numpy : 0.044054269790649414 nb_pixel_total : 120 time to create 1 rle with old method : 0.00025177001953125 time for calcul the mask position with numpy : 0.03470134735107422 nb_pixel_total : 14813 time to create 1 rle with old method : 0.019282817840576172 time for calcul the mask position with numpy : 0.04014754295349121 nb_pixel_total : 3823 time to create 1 rle with old method : 0.005057334899902344 time for calcul the mask position with numpy : 0.04218459129333496 nb_pixel_total : 1226 time to create 1 rle with old method : 0.0019910335540771484 time for calcul the mask position with numpy : 0.03987526893615723 nb_pixel_total : 80682 time to create 1 rle with old method : 0.09110736846923828 time for calcul the mask position with numpy : 0.036392927169799805 nb_pixel_total : 7014 time to create 1 rle with old method : 0.008312702178955078 time for calcul the mask position with numpy : 0.0398564338684082 nb_pixel_total : 59058 time to create 1 rle with old method : 0.08338069915771484 time for calcul the mask position with numpy : 0.04115128517150879 nb_pixel_total : 20133 time to create 1 rle with old method : 0.026350736618041992 time for calcul the mask position with numpy : 0.041860342025756836 nb_pixel_total : 32897 time to create 1 rle with old method : 0.0387721061706543 time for calcul the mask position with numpy : 0.03570699691772461 nb_pixel_total : 35135 time to create 1 rle with old method : 0.043019771575927734 time for calcul the mask position with numpy : 0.03696775436401367 nb_pixel_total : 55638 time to create 1 rle with old method : 0.06226706504821777 time for calcul the mask position with numpy : 0.038034677505493164 nb_pixel_total : 267427 time to create 1 rle with new method : 1.0470542907714844 time for calcul the mask position with numpy : 0.045464515686035156 nb_pixel_total : 30359 time to create 1 rle with old method : 0.03458118438720703 time for calcul the mask position with numpy : 0.042971134185791016 nb_pixel_total : 135450 time to create 1 rle with old method : 0.15487027168273926 time for calcul the mask position with numpy : 0.031736135482788086 nb_pixel_total : 54035 time to create 1 rle with old method : 0.06266999244689941 time for calcul the mask position with numpy : 0.023326396942138672 nb_pixel_total : 1573 time to create 1 rle with old method : 0.0020253658294677734 time for calcul the mask position with numpy : 0.024097919464111328 nb_pixel_total : 231114 time to create 1 rle with new method : 0.8051857948303223 time for calcul the mask position with numpy : 0.029688596725463867 nb_pixel_total : 153120 time to create 1 rle with new method : 0.563122034072876 create new chi : 5.554685592651367 time to delete rle : 0.0018949508666992188 batch 1 Loaded 24 chid ids of type : 3726 Number RLEs to save : 15739 TO DO : save crop sub photo not yet done ! save time : 0.9440133571624756 nb_obj : 17 nb_hashtags : 4 time to prepare the origin masks : 12.791676998138428 time for calcul the mask position with numpy : 0.45294976234436035 nb_pixel_total : 5542318 time to create 1 rle with new method : 0.4483804702758789 time for calcul the mask position with numpy : 0.024950027465820312 nb_pixel_total : 52915 time to create 1 rle with old method : 0.07503890991210938 time for calcul the mask position with numpy : 0.027091264724731445 nb_pixel_total : 16042 time to create 1 rle with old method : 0.0202789306640625 time for calcul the mask position with numpy : 0.022684335708618164 nb_pixel_total : 3377 time to create 1 rle with old method : 0.004050731658935547 time for calcul the mask position with numpy : 0.023834228515625 nb_pixel_total : 584 time to create 1 rle with old method : 0.0009603500366210938 time for calcul the mask position with numpy : 0.030457735061645508 nb_pixel_total : 2659 time to create 1 rle with old method : 0.003709554672241211 time for calcul the mask position with numpy : 0.026169538497924805 nb_pixel_total : 2333 time to create 1 rle with old method : 0.0034265518188476562 time for calcul the mask position with numpy : 0.026380062103271484 nb_pixel_total : 3338 time to create 1 rle with old method : 0.00385284423828125 time for calcul the mask position with numpy : 0.022461891174316406 nb_pixel_total : 2520 time to create 1 rle with old method : 0.002936840057373047 time for calcul the mask position with numpy : 0.023817062377929688 nb_pixel_total : 102829 time to create 1 rle with old method : 0.1146240234375 time for calcul the mask position with numpy : 0.021954774856567383 nb_pixel_total : 10228 time to create 1 rle with old method : 0.012334346771240234 time for calcul the mask position with numpy : 0.024755001068115234 nb_pixel_total : 14671 time to create 1 rle with old method : 0.022192716598510742 time for calcul the mask position with numpy : 0.02873539924621582 nb_pixel_total : 189 time to create 1 rle with old method : 0.00039458274841308594 time for calcul the mask position with numpy : 0.029554367065429688 nb_pixel_total : 37176 time to create 1 rle with old method : 0.12551498413085938 time for calcul the mask position with numpy : 0.030005931854248047 nb_pixel_total : 34416 time to create 1 rle with old method : 0.061482906341552734 time for calcul the mask position with numpy : 0.034206390380859375 nb_pixel_total : 134431 time to create 1 rle with old method : 0.24734711647033691 time for calcul the mask position with numpy : 0.04691600799560547 nb_pixel_total : 1087185 time to create 1 rle with new method : 1.4761557579040527 time for calcul the mask position with numpy : 0.024240732192993164 nb_pixel_total : 3029 time to create 1 rle with old method : 0.0036132335662841797 create new chi : 3.6193034648895264 time to delete rle : 0.0012729167938232422 batch 1 Loaded 18 chid ids of type : 3726 Number RLEs to save : 12360 TO DO : save crop sub photo not yet done ! save time : 0.8417413234710693 nb_obj : 13 nb_hashtags : 3 time to prepare the origin masks : 8.301244020462036 time for calcul the mask position with numpy : 0.4724159240722656 nb_pixel_total : 3999824 time to create 1 rle with new method : 0.991248607635498 time for calcul the mask position with numpy : 0.027959823608398438 nb_pixel_total : 338175 time to create 1 rle with new method : 0.8246743679046631 time for calcul the mask position with numpy : 0.025336265563964844 nb_pixel_total : 211743 time to create 1 rle with new method : 1.0638694763183594 time for calcul the mask position with numpy : 0.024325847625732422 nb_pixel_total : 77785 time to create 1 rle with old method : 0.08774089813232422 time for calcul the mask position with numpy : 0.0252993106842041 nb_pixel_total : 38031 time to create 1 rle with old method : 0.04454541206359863 time for calcul the mask position with numpy : 0.02997565269470215 nb_pixel_total : 60508 time to create 1 rle with old method : 0.06864094734191895 time for calcul the mask position with numpy : 0.02882528305053711 nb_pixel_total : 1891 time to create 1 rle with old method : 0.002887248992919922 time for calcul the mask position with numpy : 0.029912710189819336 nb_pixel_total : 6693 time to create 1 rle with old method : 0.007852554321289062 time for calcul the mask position with numpy : 0.025449037551879883 nb_pixel_total : 3935 time to create 1 rle with old method : 0.0047948360443115234 time for calcul the mask position with numpy : 0.02704644203186035 nb_pixel_total : 62437 time to create 1 rle with old method : 0.07151007652282715 time for calcul the mask position with numpy : 0.06411361694335938 nb_pixel_total : 2057468 time to create 1 rle with new method : 0.452495813369751 time for calcul the mask position with numpy : 0.034056663513183594 nb_pixel_total : 2549 time to create 1 rle with old method : 0.003112316131591797 time for calcul the mask position with numpy : 0.034165143966674805 nb_pixel_total : 120506 time to create 1 rle with old method : 0.15371394157409668 time for calcul the mask position with numpy : 0.02550339698791504 nb_pixel_total : 68695 time to create 1 rle with old method : 0.09821176528930664 create new chi : 4.877664089202881 time to delete rle : 0.001680135726928711 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 13642 TO DO : save crop sub photo not yet done ! save time : 1.7112324237823486 nb_obj : 21 nb_hashtags : 6 time to prepare the origin masks : 16.382770538330078 time for calcul the mask position with numpy : 0.2991158962249756 nb_pixel_total : 3382892 time to create 1 rle with new method : 0.3691368103027344 time for calcul the mask position with numpy : 0.04325151443481445 nb_pixel_total : 22450 time to create 1 rle with old method : 0.03509116172790527 time for calcul the mask position with numpy : 0.030749082565307617 nb_pixel_total : 96371 time to create 1 rle with old method : 0.12559056282043457 time for calcul the mask position with numpy : 0.03475618362426758 nb_pixel_total : 580034 time to create 1 rle with new method : 0.36159753799438477 time for calcul the mask position with numpy : 0.02412891387939453 nb_pixel_total : 9274 time to create 1 rle with old method : 0.010085105895996094 time for calcul the mask position with numpy : 0.02406167984008789 nb_pixel_total : 8619 time to create 1 rle with old method : 0.009567022323608398 time for calcul the mask position with numpy : 0.025238752365112305 nb_pixel_total : 23562 time to create 1 rle with old method : 0.02762770652770996 time for calcul the mask position with numpy : 0.02610039710998535 nb_pixel_total : 10316 time to create 1 rle with old method : 0.011474609375 time for calcul the mask position with numpy : 0.024644851684570312 nb_pixel_total : 101155 time to create 1 rle with old method : 0.10906553268432617 time for calcul the mask position with numpy : 0.035512447357177734 nb_pixel_total : 215247 time to create 1 rle with new method : 0.3958137035369873 time for calcul the mask position with numpy : 0.03156566619873047 nb_pixel_total : 520880 time to create 1 rle with new method : 0.4456315040588379 time for calcul the mask position with numpy : 0.028160572052001953 nb_pixel_total : 267960 time to create 1 rle with new method : 0.33306074142456055 time for calcul the mask position with numpy : 0.03217315673828125 nb_pixel_total : 534086 time to create 1 rle with new method : 0.3992042541503906 time for calcul the mask position with numpy : 0.024106264114379883 nb_pixel_total : 3998 time to create 1 rle with old method : 0.004555940628051758 time for calcul the mask position with numpy : 0.026141881942749023 nb_pixel_total : 8043 time to create 1 rle with old method : 0.0724329948425293 time for calcul the mask position with numpy : 0.027529478073120117 nb_pixel_total : 38307 time to create 1 rle with old method : 0.04108786582946777 time for calcul the mask position with numpy : 0.03763151168823242 nb_pixel_total : 887012 time to create 1 rle with new method : 0.3285205364227295 time for calcul the mask position with numpy : 0.027300596237182617 nb_pixel_total : 82522 time to create 1 rle with old method : 0.10074234008789062 time for calcul the mask position with numpy : 0.02830028533935547 nb_pixel_total : 5670 time to create 1 rle with old method : 0.0069293975830078125 time for calcul the mask position with numpy : 0.028354406356811523 nb_pixel_total : 9324 time to create 1 rle with old method : 0.011403322219848633 time for calcul the mask position with numpy : 0.029811859130859375 nb_pixel_total : 220454 time to create 1 rle with new method : 0.5272631645202637 time for calcul the mask position with numpy : 0.04875063896179199 nb_pixel_total : 22064 time to create 1 rle with old method : 0.025925397872924805 create new chi : 4.927926063537598 time to delete rle : 0.00257110595703125 batch 1 Loaded 22 chid ids of type : 3726 Number RLEs to save : 26525 TO DO : save crop sub photo not yet done ! save time : 1.8014733791351318 nb_obj : 10 nb_hashtags : 4 time to prepare the origin masks : 11.22305965423584 time for calcul the mask position with numpy : 0.6237063407897949 nb_pixel_total : 6650897 time to create 1 rle with new method : 0.6506216526031494 time for calcul the mask position with numpy : 0.028926610946655273 nb_pixel_total : 4347 time to create 1 rle with old method : 0.0052525997161865234 time for calcul the mask position with numpy : 0.026187896728515625 nb_pixel_total : 26784 time to create 1 rle with old method : 0.02992725372314453 time for calcul the mask position with numpy : 0.028510332107543945 nb_pixel_total : 3040 time to create 1 rle with old method : 0.0035827159881591797 time for calcul the mask position with numpy : 0.025596141815185547 nb_pixel_total : 13302 time to create 1 rle with old method : 0.01569080352783203 time for calcul the mask position with numpy : 0.022548675537109375 nb_pixel_total : 5435 time to create 1 rle with old method : 0.006240129470825195 time for calcul the mask position with numpy : 0.025033235549926758 nb_pixel_total : 203942 time to create 1 rle with new method : 0.4701573848724365 time for calcul the mask position with numpy : 0.02458930015563965 nb_pixel_total : 10355 time to create 1 rle with old method : 0.012107133865356445 time for calcul the mask position with numpy : 0.02620387077331543 nb_pixel_total : 1557 time to create 1 rle with old method : 0.0023734569549560547 time for calcul the mask position with numpy : 0.025897741317749023 nb_pixel_total : 2396 time to create 1 rle with old method : 0.0028929710388183594 time for calcul the mask position with numpy : 0.04107213020324707 nb_pixel_total : 128185 time to create 1 rle with old method : 0.14093518257141113 create new chi : 2.317986249923706 time to delete rle : 0.0008606910705566406 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 6008 TO DO : save crop sub photo not yet done ! save time : 0.39772653579711914 nb_obj : 14 nb_hashtags : 4 time to prepare the origin masks : 13.485128402709961 time for calcul the mask position with numpy : 0.4137415885925293 nb_pixel_total : 6521155 time to create 1 rle with new method : 1.4707672595977783 time for calcul the mask position with numpy : 0.029555797576904297 nb_pixel_total : 29274 time to create 1 rle with old method : 0.034288644790649414 time for calcul the mask position with numpy : 0.033585548400878906 nb_pixel_total : 7355 time to create 1 rle with old method : 0.009341955184936523 time for calcul the mask position with numpy : 0.02965235710144043 nb_pixel_total : 574 time to create 1 rle with old method : 0.0007600784301757812 time for calcul the mask position with numpy : 0.035132646560668945 nb_pixel_total : 10922 time to create 1 rle with old method : 0.012287378311157227 time for calcul the mask position with numpy : 0.02557682991027832 nb_pixel_total : 3053 time to create 1 rle with old method : 0.0035457611083984375 time for calcul the mask position with numpy : 0.03178524971008301 nb_pixel_total : 4512 time to create 1 rle with old method : 0.005688190460205078 time for calcul the mask position with numpy : 0.038198232650756836 nb_pixel_total : 14381 time to create 1 rle with old method : 0.016298770904541016 time for calcul the mask position with numpy : 0.02895665168762207 nb_pixel_total : 22794 time to create 1 rle with old method : 0.0250704288482666 time for calcul the mask position with numpy : 0.03132462501525879 nb_pixel_total : 17169 time to create 1 rle with old method : 0.019109487533569336 time for calcul the mask position with numpy : 0.029142379760742188 nb_pixel_total : 313903 time to create 1 rle with new method : 0.8544204235076904 time for calcul the mask position with numpy : 0.030670881271362305 nb_pixel_total : 2420 time to create 1 rle with old method : 0.0028655529022216797 time for calcul the mask position with numpy : 0.03262758255004883 nb_pixel_total : 14088 time to create 1 rle with old method : 0.015987634658813477 time for calcul the mask position with numpy : 0.036389827728271484 nb_pixel_total : 29715 time to create 1 rle with old method : 0.03476309776306152 time for calcul the mask position with numpy : 0.027977943420410156 nb_pixel_total : 58925 time to create 1 rle with old method : 0.07285141944885254 create new chi : 3.508805274963379 time to delete rle : 0.001566171646118164 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 8613 TO DO : save crop sub photo not yet done ! save time : 0.5433285236358643 nb_obj : 8 nb_hashtags : 3 time to prepare the origin masks : 7.208885431289673 time for calcul the mask position with numpy : 0.24105143547058105 nb_pixel_total : 5026524 time to create 1 rle with new method : 0.47457456588745117 time for calcul the mask position with numpy : 0.024830102920532227 nb_pixel_total : 4924 time to create 1 rle with old method : 0.006915569305419922 time for calcul the mask position with numpy : 0.03066706657409668 nb_pixel_total : 450244 time to create 1 rle with new method : 0.6364600658416748 time for calcul the mask position with numpy : 0.024376630783081055 nb_pixel_total : 31072 time to create 1 rle with old method : 0.04887843132019043 time for calcul the mask position with numpy : 0.02603435516357422 nb_pixel_total : 41905 time to create 1 rle with old method : 0.05298042297363281 time for calcul the mask position with numpy : 0.029450178146362305 nb_pixel_total : 160105 time to create 1 rle with new method : 0.6360001564025879 time for calcul the mask position with numpy : 0.0369715690612793 nb_pixel_total : 986475 time to create 1 rle with new method : 0.38613462448120117 time for calcul the mask position with numpy : 0.029143333435058594 nb_pixel_total : 280278 time to create 1 rle with new method : 1.7519547939300537 time for calcul the mask position with numpy : 0.023560047149658203 nb_pixel_total : 68713 time to create 1 rle with old method : 0.07776117324829102 create new chi : 4.675062894821167 time to delete rle : 0.0010867118835449219 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 11475 TO DO : save crop sub photo not yet done ! save time : 0.8369946479797363 map_output_result : {1336884922: (0.0, 'Should be the crop_list due to order', 0.0), 1336884917: (0.0, 'Should be the crop_list due to order', 0.0), 1336867305: (0.0, 'Should be the crop_list due to order', 0.0), 1336867303: (0.0, 'Should be the crop_list due to order', 0.0), 1336867300: (0.0, 'Should be the crop_list due to order', 0.0), 1336867297: (0.0, 'Should be the crop_list due to order', 0.0), 1336867290: (0.0, 'Should be the crop_list due to order', 0.0), 1336867257: (0.0, 'Should be the crop_list due to order', 0.0), 1336867251: (0.0, 'Should be the crop_list due to order', 0.0), 1336867238: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1336884922, 1336884917, 1336867305, 1336867303, 1336867300, 1336867297, 1336867290, 1336867257, 1336867251, 1336867238] Looping around the photos to save general results len do output : 10 /1336884922.Didn't retrieve data . /1336884917.Didn't retrieve data . /1336867305.Didn't retrieve data . /1336867303.Didn't retrieve data . /1336867300.Didn't retrieve data . /1336867297.Didn't retrieve data . /1336867290.Didn't retrieve data . /1336867257.Didn't retrieve data . /1336867251.Didn't retrieve data . /1336867238.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884922', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884917', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867305', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867303', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867300', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867297', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867290', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867257', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867251', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867238', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01982402801513672 save_final save missing photos in datou_result : time spend for datou_step_exec : 178.07708144187927 time spend to save output : 0.020506858825683594 total time spend for step 5 : 178.09758830070496 step6:crop_condition Tue Feb 11 21:27:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 186 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305678_72211 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.7406578063964844 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 4869462 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305687_72211 we have uploaded 9 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.3909168243408203 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305692_72211 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.3884150981903076 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305695_72211 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.148357629776001 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305699_72211 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.7643585205078125 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305704_72211 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.024853229522705 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1336884922, 1336884917, 1336867305, 1336867303, 1336867300, 1336867297, 1336867290, 1336867257, 1336867251, 1336867238] Looping around the photos to save general results len do output : 33 /1336911139Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911140Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911141Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911142Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911143Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911144Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911145Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911146Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911164Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911165Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911167Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911171Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911174Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911175Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911176Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911177Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911178Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911181Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911182Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911183Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911184Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911185Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911186Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911187Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911188Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911189Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911190Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911191Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911192Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911193Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911194Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911209Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336911212Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884922', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884917', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867305', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867303', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867300', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867297', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867290', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867257', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867251', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867238', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 109 time used for this insertion : 0.04384016990661621 save_final save missing photos in datou_result : time spend for datou_step_exec : 34.228387117385864 time spend to save output : 0.06346893310546875 total time spend for step 6 : 34.29185605049133 step7:ventilate_hashtags_in_portfolio Tue Feb 11 21:28:26 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20461712 get user id for portfolio 20461712 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461712 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('autre_refus','Carton_gris','environnement','metal','papier','Teint_Dans_La_Masse','plastique','flou','cartonnette','Carton_brun','mal_croppe','kraft')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461712 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('autre_refus','Carton_gris','environnement','metal','papier','Teint_Dans_La_Masse','plastique','flou','cartonnette','Carton_brun','mal_croppe','kraft')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20461867,20461868,20461869,20461870,20461871,20461872,20461873,20461874,20461875,20461876,20461877,20461878?tags=metal,flou,plastique,Teint_Dans_La_Masse,Carton_brun,mal_croppe,papier,autre_refus,cartonnette,kraft,Carton_gris,environnement Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1336884922, 1336884917, 1336867305, 1336867303, 1336867300, 1336867297, 1336867290, 1336867257, 1336867251, 1336867238] Looping around the photos to save general results len do output : 1 /20461712. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884922', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884917', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867305', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867303', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867300', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867297', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867290', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867257', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867251', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867238', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014951467514038086 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.6437301635742188 time spend to save output : 0.015239238739013672 total time spend for step 7 : 0.6589694023132324 step8:final Tue Feb 11 21:28:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1336884922: ('0.10233605795847883',), 1336884917: ('0.10233605795847883',), 1336867305: ('0.10233605795847883',), 1336867303: ('0.10233605795847883',), 1336867300: ('0.10233605795847883',), 1336867297: ('0.10233605795847883',), 1336867290: ('0.10233605795847883',), 1336867257: ('0.10233605795847883',), 1336867251: ('0.10233605795847883',), 1336867238: ('0.10233605795847883',)} new output for save of step final : {1336884922: ('0.10233605795847883',), 1336884917: ('0.10233605795847883',), 1336867305: ('0.10233605795847883',), 1336867303: ('0.10233605795847883',), 1336867300: ('0.10233605795847883',), 1336867297: ('0.10233605795847883',), 1336867290: ('0.10233605795847883',), 1336867257: ('0.10233605795847883',), 1336867251: ('0.10233605795847883',), 1336867238: ('0.10233605795847883',)} [1336884922, 1336884917, 1336867305, 1336867303, 1336867300, 1336867297, 1336867290, 1336867257, 1336867251, 1336867238] Looping around the photos to save general results len do output : 10 /1336884922.Didn't retrieve data . /1336884917.Didn't retrieve data . /1336867305.Didn't retrieve data . /1336867303.Didn't retrieve data . /1336867300.Didn't retrieve data . /1336867297.Didn't retrieve data . /1336867290.Didn't retrieve data . /1336867257.Didn't retrieve data . /1336867251.Didn't retrieve data . /1336867238.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884922', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884917', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867305', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867303', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867300', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867297', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867290', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867257', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867251', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867238', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.016406774520874023 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.1089944839477539 time spend to save output : 0.017007112503051758 total time spend for step 8 : 0.12600159645080566 step9:velours_tree Tue Feb 11 21:28:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.12029790878295898 time spend to save output : 4.0531158447265625e-05 total time spend for step 9 : 0.12033843994140625 step10:send_mail_cod Tue Feb 11 21:28:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20461712_11-02-2025_21_28_27.pdf 20461867 change filename to text .imagette204618671739305707 20461868 imagette204618681739305707 20461869 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204618691739305707 20461872 imagette204618721739305708 20461874 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204618741739305708 20461875 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204618751739305710 20461876 imagette204618761739305710 20461870 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204618701739305711 20461871 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204618711739305711 20461877 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204618771739305712 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20461712 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20461867,20461868,20461869,20461870,20461871,20461872,20461873,20461874,20461875,20461876,20461877,20461878?tags=metal,flou,plastique,Teint_Dans_La_Masse,Carton_brun,mal_croppe,papier,autre_refus,cartonnette,kraft,Carton_gris,environnement your option no_mail is active, we will not send the real mail to your client args[1336884922] : ('0.10233605795847883',) no score found for photo 1336884922 apple ('0.10233605795847883',) We are sending mail with results at report@fotonower.com args[1336884917] : ('0.10233605795847883',) no score found for photo 1336884917 apple ('0.10233605795847883',) We are sending mail with results at report@fotonower.com args[1336867305] : ('0.10233605795847883',) no score found for photo 1336867305 apple ('0.10233605795847883',) We are sending mail with results at report@fotonower.com args[1336867303] : ('0.10233605795847883',) no score found for photo 1336867303 apple ('0.10233605795847883',) We are sending mail with results at report@fotonower.com args[1336867300] : ('0.10233605795847883',) no score found for photo 1336867300 apple ('0.10233605795847883',) We are sending mail with results at report@fotonower.com args[1336867297] : ('0.10233605795847883',) no score found for photo 1336867297 apple ('0.10233605795847883',) We are sending mail with results at report@fotonower.com args[1336867290] : ('0.10233605795847883',) no score found for photo 1336867290 apple ('0.10233605795847883',) We are sending mail with results at report@fotonower.com args[1336867257] : ('0.10233605795847883',) no score found for photo 1336867257 apple ('0.10233605795847883',) We are sending mail with results at report@fotonower.com args[1336867251] : ('0.10233605795847883',) no score found for photo 1336867251 apple ('0.10233605795847883',) We are sending mail with results at report@fotonower.com args[1336867238] : ('0.10233605795847883',) no score found for photo 1336867238 apple ('0.10233605795847883',) We are sending mail with results at report@fotonower.com refus_total : 0.10233605795847883 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20461712 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1336867303,1336885647,1336885641,1336885604,1336885562,1336885512,1336885502,1336884922,1336867305,1336866945,1336867300,1336867297,1336867257,1336867251,1336867056,1336867047,1336867045,1336867043,1336867040,1336867053) Found this number of photos: 20 begin to download photo : 1336867303 begin to download photo : 1336885512 begin to download photo : 1336867300 begin to download photo : 1336867047 download finish for photo 1336867303 begin to download photo : 1336885647 download finish for photo 1336885512 begin to download photo : 1336885502 download finish for photo 1336867047 begin to download photo : 1336867045 download finish for photo 1336867300 begin to download photo : 1336867297 download finish for photo 1336885502 begin to download photo : 1336884922 download finish for photo 1336867297 begin to download photo : 1336867257 download finish for photo 1336885647 begin to download photo : 1336885641 download finish for photo 1336867045 begin to download photo : 1336867043 download finish for photo 1336884922 begin to download photo : 1336867305 download finish for photo 1336885641 begin to download photo : 1336885604 download finish for photo 1336867257 begin to download photo : 1336867251 download finish for photo 1336867043 begin to download photo : 1336867040 download finish for photo 1336867305 begin to download photo : 1336866945 download finish for photo 1336885604 begin to download photo : 1336885562 download finish for photo 1336867251 begin to download photo : 1336867056 download finish for photo 1336867040 begin to download photo : 1336867053 download finish for photo 1336866945 download finish for photo 1336885562 download finish for photo 1336867056 download finish for photo 1336867053 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461712_11-02-2025_21_28_27.pdf results_Auto_P20461712_11-02-2025_21_28_27.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461712_11-02-2025_21_28_27.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20461712','results_Auto_P20461712_11-02-2025_21_28_27.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461712_11-02-2025_21_28_27.pdf','pdf','','1.33','0.10233605795847883') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1336884922, 1336884917, 1336867305, 1336867303, 1336867300, 1336867297, 1336867290, 1336867257, 1336867251, 1336867238] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884922', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884917', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867305', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867303', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867300', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867297', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867290', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867257', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867251', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867238', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.01824331283569336 save_final save missing photos in datou_result : time spend for datou_step_exec : 10.262616157531738 time spend to save output : 0.018500089645385742 total time spend for step 10 : 10.281116247177124 step11:split_time_score Tue Feb 11 21:28:37 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('17', 28),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 11022025 20461712 Nombre de photos uploadées : 28 / 23040 (0%) 11022025 20461712 Nombre de photos taguées (types de déchets): 0 / 28 (0%) 11022025 20461712 Nombre de photos taguées (volume) : 0 / 28 (0%) elapsed_time : load_data_split_time_score 3.814697265625e-06 elapsed_time : order_list_meta_photo_and_scores 9.298324584960938e-06 ???????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0013768672943115234 elapsed_time : insert_dashboard_record_day_entry 0.033301353454589844 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461688 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461689 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461690 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461691 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461692 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461693 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461694 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461695 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461696 order by id desc limit 1 Qualite : 0.08017768812741913 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461697_11-02-2025_21_21_00.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461697 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461697 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461698 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461699 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461700 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461701 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461702 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461703 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461704 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461705 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461706 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461707 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461708 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461709 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461710 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461711 order by id desc limit 1 Qualite : 0.11839248806077622 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461712_11-02-2025_21_28_27.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461712 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461712 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461713 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461714 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461715 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461716 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461717 order by id desc limit 1 Qualite : 0.02953539912490668 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461718_11-02-2025_21_26_33.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461718 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461718 AND mptpi.`type`=4207 To do Qualite : 0.041030917312523105 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461719_11-02-2025_21_22_24.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461719 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461719 AND mptpi.`type`=4207 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'11022025': {'nb_upload': 28, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1336884922, 1336884917, 1336867305, 1336867303, 1336867300, 1336867297, 1336867290, 1336867257, 1336867251, 1336867238] Looping around the photos to save general results len do output : 1 /20461712Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884922', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336884917', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867305', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867303', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867300', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867297', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867290', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867257', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867251', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', None, '1336867238', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.016236066818237305 save_final save missing photos in datou_result : time spend for datou_step_exec : 24.35940957069397 time spend to save output : 0.016445398330688477 total time spend for step 11 : 24.375854969024658 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 202.81user 152.63system 6:36.30elapsed 89%CPU (0avgtext+0avgdata 4583788maxresident)k 2619048inputs+186072outputs (12454major+13652779minor)pagefaults 0swaps