python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/datou.py -j batch_current -a 4433' -s traitement_sts -M 0 -S 0 -U 100,80,95 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/home/admin/workarea/git/apy', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 55108 load datou : 4433 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4433, datou_cur_ids : ['3784106'] with mtr_portfolio_ids : ['27267728'] and first list_photo_ids : [] new path : /proc/55108/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, brightness, blur_detection, crop_condition, thcl, tfhub_classification2, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score over limit max, limiting to limit_max 15 list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 8 ; length of list_pids : 8 ; length of list_args : 8 time to download the photos : 2.988471269607544 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 14 step1:mask_detect Fri Sep 26 11:20:15 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10582 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-09-26 11:20:19.397626: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-09-26 11:20:19.428567: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-09-26 11:20:19.430606: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fb854000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-09-26 11:20:19.430665: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-09-26 11:20:19.434973: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-09-26 11:20:19.613675: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1277e790 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-09-26 11:20:19.613756: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-09-26 11:20:19.615961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-26 11:20:19.617762: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-26 11:20:19.626700: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-26 11:20:19.641351: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-26 11:20:19.643848: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-26 11:20:19.656659: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-26 11:20:19.659303: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-26 11:20:19.680518: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-26 11:20:19.682346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-26 11:20:19.683138: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-26 11:20:19.684282: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-26 11:20:19.684302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-26 11:20:19.684314: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-26 11:20:19.686650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9801 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-09-26 11:20:20.109359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-26 11:20:20.109446: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-26 11:20:20.109464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-26 11:20:20.109480: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-26 11:20:20.109495: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-26 11:20:20.109509: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-26 11:20:20.109523: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-26 11:20:20.109538: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-26 11:20:20.110748: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-26 11:20:20.111820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-26 11:20:20.111852: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-26 11:20:20.111873: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-26 11:20:20.111890: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-26 11:20:20.111906: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-26 11:20:20.111922: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-26 11:20:20.111937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-26 11:20:20.111953: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-26 11:20:20.113239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-26 11:20:20.113278: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-26 11:20:20.113288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-26 11:20:20.113295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-26 11:20:20.114623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9801 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-09-26 11:20:33.194975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-26 11:20:33.529174: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 8 NEW PHOTO Processing 1 images image shape: (3040, 4056, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 4056.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (3040, 4056, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 4056.00000 nb d'objets trouves : 21 NEW PHOTO Processing 1 images image shape: (3040, 4056, 3) min: 2.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 4056.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (3040, 4056, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 4056.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (3040, 4056, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 4056.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (3040, 4056, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 4056.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (3040, 4056, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 4056.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (3040, 4056, 3) min: 5.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 4056.00000 nb d'objets trouves : 13 Detection mask done ! Trying to reset tf kernel 55315 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 409 tf kernel not reseted sub process len(results) : 8 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 8 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5698 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.006877899169921875 nb_pixel_total : 103415 time to create 1 rle with old method : 0.11907219886779785 length of segment : 328 time for calcul the mask position with numpy : 0.00318145751953125 nb_pixel_total : 231346 time to create 1 rle with new method : 0.009741544723510742 length of segment : 487 time for calcul the mask position with numpy : 0.004634857177734375 nb_pixel_total : 353986 time to create 1 rle with new method : 0.013730287551879883 length of segment : 861 time for calcul the mask position with numpy : 0.0006289482116699219 nb_pixel_total : 48354 time to create 1 rle with old method : 0.05413365364074707 length of segment : 219 time for calcul the mask position with numpy : 0.0017046928405761719 nb_pixel_total : 119884 time to create 1 rle with old method : 0.13084793090820312 length of segment : 545 time for calcul the mask position with numpy : 0.0014944076538085938 nb_pixel_total : 82333 time to create 1 rle with old method : 0.0899510383605957 length of segment : 396 time for calcul the mask position with numpy : 0.002525806427001953 nb_pixel_total : 186075 time to create 1 rle with new method : 0.008790731430053711 length of segment : 423 time for calcul the mask position with numpy : 0.06834101676940918 nb_pixel_total : 651195 time to create 1 rle with new method : 0.44373345375061035 length of segment : 2407 time for calcul the mask position with numpy : 0.0026025772094726562 nb_pixel_total : 135289 time to create 1 rle with old method : 0.1518721580505371 length of segment : 505 time for calcul the mask position with numpy : 0.00041675567626953125 nb_pixel_total : 25420 time to create 1 rle with old method : 0.028491497039794922 length of segment : 175 time for calcul the mask position with numpy : 0.004221439361572266 nb_pixel_total : 269545 time to create 1 rle with new method : 0.014531373977661133 length of segment : 710 time for calcul the mask position with numpy : 0.002686738967895508 nb_pixel_total : 193795 time to create 1 rle with new method : 0.0062808990478515625 length of segment : 581 time for calcul the mask position with numpy : 0.0010502338409423828 nb_pixel_total : 70779 time to create 1 rle with old method : 0.07872772216796875 length of segment : 329 time for calcul the mask position with numpy : 0.0006165504455566406 nb_pixel_total : 43189 time to create 1 rle with old method : 0.04879903793334961 length of segment : 212 time for calcul the mask position with numpy : 0.00010704994201660156 nb_pixel_total : 4072 time to create 1 rle with old method : 0.004911899566650391 length of segment : 54 time for calcul the mask position with numpy : 0.007150888442993164 nb_pixel_total : 488653 time to create 1 rle with new method : 0.057334184646606445 length of segment : 860 time for calcul the mask position with numpy : 0.00029659271240234375 nb_pixel_total : 13669 time to create 1 rle with old method : 0.016092538833618164 length of segment : 162 time for calcul the mask position with numpy : 0.0003719329833984375 nb_pixel_total : 20715 time to create 1 rle with old method : 0.023647069931030273 length of segment : 229 time for calcul the mask position with numpy : 0.0006718635559082031 nb_pixel_total : 41045 time to create 1 rle with old method : 0.04962301254272461 length of segment : 323 time for calcul the mask position with numpy : 0.0016629695892333984 nb_pixel_total : 67931 time to create 1 rle with old method : 0.08081793785095215 length of segment : 485 time for calcul the mask position with numpy : 0.0003757476806640625 nb_pixel_total : 18179 time to create 1 rle with old method : 0.02145099639892578 length of segment : 176 time for calcul the mask position with numpy : 0.0003204345703125 nb_pixel_total : 13770 time to create 1 rle with old method : 0.023476123809814453 length of segment : 136 time for calcul the mask position with numpy : 0.0010614395141601562 nb_pixel_total : 57243 time to create 1 rle with old method : 0.08055853843688965 length of segment : 368 time for calcul the mask position with numpy : 0.002310037612915039 nb_pixel_total : 143217 time to create 1 rle with old method : 0.16001009941101074 length of segment : 557 time for calcul the mask position with numpy : 0.007156848907470703 nb_pixel_total : 174259 time to create 1 rle with new method : 0.14929842948913574 length of segment : 447 time for calcul the mask position with numpy : 0.0018799304962158203 nb_pixel_total : 83852 time to create 1 rle with old method : 0.09367847442626953 length of segment : 505 time for calcul the mask position with numpy : 0.0008561611175537109 nb_pixel_total : 48247 time to create 1 rle with old method : 0.06262469291687012 length of segment : 265 time for calcul the mask position with numpy : 0.00046443939208984375 nb_pixel_total : 25909 time to create 1 rle with old method : 0.029616594314575195 length of segment : 292 time for calcul the mask position with numpy : 0.003928661346435547 nb_pixel_total : 320843 time to create 1 rle with new method : 0.01065516471862793 length of segment : 462 time for calcul the mask position with numpy : 0.0053331851959228516 nb_pixel_total : 255114 time to create 1 rle with new method : 0.022004127502441406 length of segment : 890 time for calcul the mask position with numpy : 0.004881381988525391 nb_pixel_total : 405179 time to create 1 rle with new method : 0.010713577270507812 length of segment : 1035 time for calcul the mask position with numpy : 0.0031430721282958984 nb_pixel_total : 150466 time to create 1 rle with new method : 0.0057621002197265625 length of segment : 587 time for calcul the mask position with numpy : 0.007920503616333008 nb_pixel_total : 423714 time to create 1 rle with new method : 0.03252053260803223 length of segment : 788 time for calcul the mask position with numpy : 0.0271303653717041 nb_pixel_total : 59237 time to create 1 rle with old method : 0.06707429885864258 length of segment : 509 time for calcul the mask position with numpy : 0.0006084442138671875 nb_pixel_total : 21143 time to create 1 rle with old method : 0.02442336082458496 length of segment : 275 time for calcul the mask position with numpy : 0.014528512954711914 nb_pixel_total : 633671 time to create 1 rle with new method : 0.05434679985046387 length of segment : 1345 time for calcul the mask position with numpy : 0.002260446548461914 nb_pixel_total : 130758 time to create 1 rle with old method : 0.14778351783752441 length of segment : 202 time for calcul the mask position with numpy : 0.0037767887115478516 nb_pixel_total : 65623 time to create 1 rle with old method : 0.08056473731994629 length of segment : 532 time for calcul the mask position with numpy : 0.00024199485778808594 nb_pixel_total : 12439 time to create 1 rle with old method : 0.014307022094726562 length of segment : 131 time for calcul the mask position with numpy : 0.03024768829345703 nb_pixel_total : 864234 time to create 1 rle with new method : 0.296114444732666 length of segment : 1904 time for calcul the mask position with numpy : 0.0009648799896240234 nb_pixel_total : 44289 time to create 1 rle with old method : 0.0496213436126709 length of segment : 196 time for calcul the mask position with numpy : 0.0005917549133300781 nb_pixel_total : 14621 time to create 1 rle with old method : 0.017879486083984375 length of segment : 306 time for calcul the mask position with numpy : 0.0034384727478027344 nb_pixel_total : 218293 time to create 1 rle with new method : 0.0073511600494384766 length of segment : 429 time for calcul the mask position with numpy : 0.0014276504516601562 nb_pixel_total : 50505 time to create 1 rle with old method : 0.05906558036804199 length of segment : 439 time for calcul the mask position with numpy : 0.0005686283111572266 nb_pixel_total : 20232 time to create 1 rle with old method : 0.02643132209777832 length of segment : 149 time for calcul the mask position with numpy : 0.004385709762573242 nb_pixel_total : 143181 time to create 1 rle with old method : 0.1851954460144043 length of segment : 555 time for calcul the mask position with numpy : 0.00036334991455078125 nb_pixel_total : 3814 time to create 1 rle with old method : 0.004137754440307617 length of segment : 121 time for calcul the mask position with numpy : 0.006921052932739258 nb_pixel_total : 330977 time to create 1 rle with new method : 0.009993553161621094 length of segment : 841 time for calcul the mask position with numpy : 0.0012669563293457031 nb_pixel_total : 41141 time to create 1 rle with old method : 0.04675126075744629 length of segment : 235 time for calcul the mask position with numpy : 0.002572298049926758 nb_pixel_total : 107517 time to create 1 rle with old method : 0.12672162055969238 length of segment : 445 time for calcul the mask position with numpy : 0.005952119827270508 nb_pixel_total : 260543 time to create 1 rle with new method : 0.00889134407043457 length of segment : 689 time for calcul the mask position with numpy : 0.0013842582702636719 nb_pixel_total : 68960 time to create 1 rle with old method : 0.07723498344421387 length of segment : 284 time for calcul the mask position with numpy : 0.007433652877807617 nb_pixel_total : 226869 time to create 1 rle with new method : 0.013161182403564453 length of segment : 927 time for calcul the mask position with numpy : 0.0039005279541015625 nb_pixel_total : 189470 time to create 1 rle with new method : 0.0076410770416259766 length of segment : 543 time for calcul the mask position with numpy : 0.00037407875061035156 nb_pixel_total : 15150 time to create 1 rle with old method : 0.01787090301513672 length of segment : 73 time for calcul the mask position with numpy : 0.0012772083282470703 nb_pixel_total : 63428 time to create 1 rle with old method : 0.07507658004760742 length of segment : 203 time for calcul the mask position with numpy : 0.0009553432464599609 nb_pixel_total : 59574 time to create 1 rle with old method : 0.07263302803039551 length of segment : 194 time for calcul the mask position with numpy : 0.001294851303100586 nb_pixel_total : 36880 time to create 1 rle with old method : 0.044777631759643555 length of segment : 372 time for calcul the mask position with numpy : 0.00042724609375 nb_pixel_total : 3962 time to create 1 rle with old method : 0.007180213928222656 length of segment : 53 time for calcul the mask position with numpy : 0.0006690025329589844 nb_pixel_total : 15070 time to create 1 rle with old method : 0.019681453704833984 length of segment : 171 time for calcul the mask position with numpy : 0.0009090900421142578 nb_pixel_total : 24816 time to create 1 rle with old method : 0.03063368797302246 length of segment : 242 time for calcul the mask position with numpy : 0.0047702789306640625 nb_pixel_total : 155750 time to create 1 rle with new method : 0.008121013641357422 length of segment : 424 time for calcul the mask position with numpy : 0.0004448890686035156 nb_pixel_total : 7950 time to create 1 rle with old method : 0.00987386703491211 length of segment : 156 time for calcul the mask position with numpy : 0.021448850631713867 nb_pixel_total : 709007 time to create 1 rle with new method : 0.07660388946533203 length of segment : 1894 time for calcul the mask position with numpy : 0.3786444664001465 nb_pixel_total : 1672130 time to create 1 rle with new method : 0.4009573459625244 length of segment : 2514 time for calcul the mask position with numpy : 0.0030808448791503906 nb_pixel_total : 147684 time to create 1 rle with old method : 0.16297388076782227 length of segment : 452 time for calcul the mask position with numpy : 0.0007479190826416016 nb_pixel_total : 15975 time to create 1 rle with old method : 0.018196582794189453 length of segment : 265 time for calcul the mask position with numpy : 0.0009603500366210938 nb_pixel_total : 33939 time to create 1 rle with old method : 0.03868222236633301 length of segment : 185 time for calcul the mask position with numpy : 0.0005242824554443359 nb_pixel_total : 13001 time to create 1 rle with old method : 0.01508188247680664 length of segment : 162 time for calcul the mask position with numpy : 0.002976655960083008 nb_pixel_total : 119789 time to create 1 rle with old method : 0.13467931747436523 length of segment : 378 time for calcul the mask position with numpy : 0.0047490596771240234 nb_pixel_total : 130264 time to create 1 rle with old method : 0.14599990844726562 length of segment : 652 time for calcul the mask position with numpy : 0.0010101795196533203 nb_pixel_total : 30502 time to create 1 rle with old method : 0.035361528396606445 length of segment : 282 time for calcul the mask position with numpy : 0.0050525665283203125 nb_pixel_total : 93211 time to create 1 rle with old method : 0.10271334648132324 length of segment : 797 time for calcul the mask position with numpy : 0.0002009868621826172 nb_pixel_total : 3828 time to create 1 rle with old method : 0.004658937454223633 length of segment : 54 time for calcul the mask position with numpy : 0.0007143020629882812 nb_pixel_total : 28028 time to create 1 rle with old method : 0.03148674964904785 length of segment : 279 time for calcul the mask position with numpy : 0.005606889724731445 nb_pixel_total : 169009 time to create 1 rle with new method : 0.040578603744506836 length of segment : 1073 time for calcul the mask position with numpy : 0.002174854278564453 nb_pixel_total : 71270 time to create 1 rle with old method : 0.10351037979125977 length of segment : 370 time for calcul the mask position with numpy : 0.0004565715789794922 nb_pixel_total : 11925 time to create 1 rle with old method : 0.013669252395629883 length of segment : 120 time for calcul the mask position with numpy : 0.002107381820678711 nb_pixel_total : 62772 time to create 1 rle with old method : 0.06965517997741699 length of segment : 404 time for calcul the mask position with numpy : 0.0007617473602294922 nb_pixel_total : 32551 time to create 1 rle with old method : 0.03799104690551758 length of segment : 112 time for calcul the mask position with numpy : 0.0007791519165039062 nb_pixel_total : 23655 time to create 1 rle with old method : 0.027114391326904297 length of segment : 181 time for calcul the mask position with numpy : 0.0007417201995849609 nb_pixel_total : 21422 time to create 1 rle with old method : 0.024682998657226562 length of segment : 293 time for calcul the mask position with numpy : 0.0007998943328857422 nb_pixel_total : 30814 time to create 1 rle with old method : 0.03549766540527344 length of segment : 173 time for calcul the mask position with numpy : 0.0007910728454589844 nb_pixel_total : 22328 time to create 1 rle with old method : 0.026177167892456055 length of segment : 285 time for calcul the mask position with numpy : 0.0023293495178222656 nb_pixel_total : 72498 time to create 1 rle with old method : 0.08188867568969727 length of segment : 411 time for calcul the mask position with numpy : 0.0014891624450683594 nb_pixel_total : 43659 time to create 1 rle with old method : 0.05178427696228027 length of segment : 647 length of segment : 0 time for calcul the mask position with numpy : 0.006262302398681641 nb_pixel_total : 215637 time to create 1 rle with new method : 0.017507553100585938 length of segment : 829 time spent for convertir_results : 22.93068265914917 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 94 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 42556 save missing photos in datou_result : time spend for datou_step_exec : 74.25922679901123 time spend to save output : 2.317213535308838 total time spend for step 1 : 76.57644033432007 step2:brightness Fri Sep 26 11:21:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure inside step calcul brightness toutes les photos sont déjà traitées, on saute les calculs Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 8 time used for this insertion : 0.00887441635131836 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 8 time used for this insertion : 0.008928298950195312 save missing photos in datou_result : time spend for datou_step_exec : 0.029011011123657227 time spend to save output : 0.022574663162231445 total time spend for step 2 : 0.05158567428588867 step3:blur_detection Fri Sep 26 11:21:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure inside step blur_detection toutes les photos sont déjà traitées, on saute les calculs Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 8 time used for this insertion : 0.008724212646484375 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 8 time used for this insertion : 0.009035348892211914 save missing photos in datou_result : time spend for datou_step_exec : 0.021392345428466797 time spend to save output : 0.02254772186279297 total time spend for step 3 : 0.043940067291259766 step4:crop_condition Fri Sep 26 11:21:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 8 ! batch 1 Loaded 94 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 12 About to insert : list_path_to_insert length 12 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 28 About to insert : list_path_to_insert length 28 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1385110088, 1385110084, 1385110083, 1385110080, 1385106334, 1385106332, 1385106331, 1385106321] Looping around the photos to save general results len do output : 69 /-3969766785Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766798Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766796Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766794Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766801Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766817Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766812Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3976635422Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766825Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766820Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766863Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766861Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766809Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766810Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766828Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766832Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766830Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766840Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766869Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766860Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766855Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766856Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766786Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766789Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766797Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766813Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766816Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766819Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766815Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766821Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766834Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766849Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766857Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766791Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766788Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766790Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766799Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766802Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766806Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766795Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766804Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766805Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766811Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766822Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766823Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766831Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766829Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766833Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766847Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766845Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766844Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766842Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3976635424Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766854Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766852Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766853Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766865Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766862Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766867Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766868Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766866Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766793Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766787Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3976635421Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766836Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766835Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766837Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766841Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3969766872Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110088', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110084', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110083', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110080', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106334', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106332', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106331', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106321', None, None, None, None, None, '3784106') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 215 time used for this insertion : 0.034650325775146484 save_final save missing photos in datou_result : time spend for datou_step_exec : 55.080188274383545 time spend to save output : 0.5513138771057129 total time spend for step 4 : 55.63150215148926 step5:thcl Fri Sep 26 11:22:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 1 we are using the classfication for multi_thcl [2456] time to import caffe and check if the image exist : 0.017592668533325195 time to convert the images to numpy array : 0.07290148735046387 time to import caffe and check if the image exist : 0.015370368957519531 time to convert the images to numpy array : 0.0988471508026123 time to import caffe and check if the image exist : 0.014938116073608398 time to convert the images to numpy array : 0.3719804286956787 time to import caffe and check if the image exist : 0.02071094512939453 time to convert the images to numpy array : 0.4597187042236328 time to import caffe and check if the image exist : 0.010070085525512695 time to convert the images to numpy array : 0.7536871433258057 time to import caffe and check if the image exist : 0.02006816864013672 time to convert the images to numpy array : 0.7978901863098145 time to import caffe and check if the image exist : 0.012967824935913086 time to convert the images to numpy array : 0.8155460357666016 time to import caffe and check if the image exist : 0.01640486717224121 time to convert the images to numpy array : 0.9130549430847168 time to import caffe and check if the image exist : 0.017023801803588867 time to convert the images to numpy array : 0.9229283332824707 time to import caffe and check if the image exist : 0.014811277389526367 time to convert the images to numpy array : 0.9353046417236328 total time to convert the images to numpy array : 1.7107582092285156 list photo_ids error: [] list photo_ids correct : [-3969766845, -3969766844, -3969766842, -3976635424, -3969766854, -3969766852, -3969766853, -3969766811, -3969766822, -3969766823, -3969766831, -3969766829, -3969766833, -3969766847, -3969766790, -3969766799, -3969766802, -3969766806, -3969766795, -3969766804, -3969766805, -3969766865, -3969766862, -3969766867, -3969766868, -3969766866, -3969766793, -3969766787, -3969766785, -3969766798, -3969766796, -3969766794, -3969766801, -3969766817, -3969766812, -3969766856, -3969766786, -3969766789, -3969766797, -3969766813, -3969766816, -3969766819, -3969766815, -3969766821, -3969766834, -3969766849, -3969766857, -3969766791, -3969766788, -3969766828, -3969766832, -3969766830, -3969766840, -3969766869, -3969766860, -3969766855, -3976635422, -3969766825, -3969766820, -3969766863, -3969766861, -3969766809, -3969766810, -3976635421, -3969766836, -3969766835, -3969766837, -3969766841, -3969766872] number of photos to traite : 69 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10582 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/:/home/admin/workarea/git/apy/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10360 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.2889008522033691 time used to do the prediction : 0.21025824546813965 we don't save the descriptors for this thcl 2456 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 3.361701965332031e-05 save missing photos in datou_result : time spend for datou_step_exec : 7.3214147090911865 time spend to save output : 0.00038933753967285156 total time spend for step 5 : 7.321804046630859 step6:tfhub_classification2 Fri Sep 26 11:22:35 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou_step TFHub with tf2 ! we are using the classfication for only one thcl 3501 begin to check gpu status inside check gpu memory 2025-09-26 11:22:38.853616: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-09-26 11:22:38.855868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-26 11:22:38.855956: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-26 11:22:38.856037: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-26 11:22:38.876408: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-26 11:22:38.876516: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-26 11:22:38.910797: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-26 11:22:38.915796: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-26 11:22:38.972143: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-26 11:22:38.974009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-26 11:22:38.974619: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-09-26 11:22:39.004562: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-09-26 11:22:39.006504: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fb7a8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-09-26 11:22:39.006547: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-09-26 11:22:39.011745: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1e075b40 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-09-26 11:22:39.011776: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-09-26 11:22:39.013202: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-26 11:22:39.013280: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-26 11:22:39.013312: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-26 11:22:39.013400: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-26 11:22:39.013442: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-26 11:22:39.013498: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-26 11:22:39.013548: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-26 11:22:39.013612: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-26 11:22:39.015879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-26 11:22:39.016319: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-26 11:22:39.016381: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-26 11:22:39.016401: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-26 11:22:39.016414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-26 11:22:39.018148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 10358 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3501 To do loadFromThcl(), then load ParamDescType : thcl3501 thcls : [{'id': 3501, 'mtr_user_id': 31, 'name': 'learn_papier_mixte_centre_tf_b4_220223_3', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '9087724,9087727,9087725,9087729,9087726,9087723,9087722,9087720,9087721,9087728', 'photo_hashtag_type': 4539, 'photo_desc_type': 5759, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 3501, 'mtr_user_id': 31, 'name': 'learn_papier_mixte_centre_tf_b4_220223_3', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '9087724,9087727,9087725,9087729,9087726,9087723,9087722,9087720,9087721,9087728', 'photo_hashtag_type': 4539, 'photo_desc_type': 5759, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5759 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5759, 'learn_papier_mixte_centre_tf_b4_220223_3', 1792, 1792, 'learn_papier_mixte_centre_tf_b4_220223_3', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 2, 22, 13, 54, 32), datetime.datetime(2023, 2, 22, 13, 54, 32)) model_name : learn_papier_mixte_centre_tf_b4_220223_3 model_param file didn't exist model_name : learn_papier_mixte_centre_tf_b4_220223_3 model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_Carton_brun.jpg', 'Precision_Recall_Carton_gris.jpg', 'Precision_Recall_Teint_Dans_La_Masse.jpg', 'Precision_Recall_autre_refus.jpg', 'Precision_Recall_cartonnette.jpg', 'Precision_Recall_environnement.jpg', 'Precision_Recall_kraft.jpg', 'Precision_Recall_metal.jpg', 'Precision_Recall_papier.jpg', 'Precision_Recall_plastique.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_Carton_brun.jpg', 'Precision_Recall_Carton_gris.jpg', 'Precision_Recall_Teint_Dans_La_Masse.jpg', 'Precision_Recall_autre_refus.jpg', 'Precision_Recall_cartonnette.jpg', 'Precision_Recall_environnement.jpg', 'Precision_Recall_kraft.jpg', 'Precision_Recall_metal.jpg', 'Precision_Recall_papier.jpg', 'Precision_Recall_plastique.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3 /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/Confusion_Matrix.png size_local : 121269 size in s3 : 121269 create time local : 2023-03-03 12:26:14 create time in s3 : 2023-02-22 12:54:17 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/Precision_Recall_Carton_brun.jpg size_local : 76104 size in s3 : 76104 create time local : 2023-03-03 12:26:14 create time in s3 : 2023-02-22 12:54:31 Precision_Recall_Carton_brun.jpg already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/Precision_Recall_Carton_gris.jpg size_local : 82912 size in s3 : 82912 create time local : 2023-03-03 12:26:14 create time in s3 : 2023-02-22 12:54:31 Precision_Recall_Carton_gris.jpg already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/Precision_Recall_Teint_Dans_La_Masse.jpg size_local : 75423 size in s3 : 75423 create time local : 2023-03-03 12:26:14 create time in s3 : 2023-02-22 12:54:25 Precision_Recall_Teint_Dans_La_Masse.jpg already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/Precision_Recall_autre_refus.jpg size_local : 78883 size in s3 : 78883 create time local : 2023-03-03 12:26:15 create time in s3 : 2023-02-22 12:54:31 Precision_Recall_autre_refus.jpg already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/Precision_Recall_cartonnette.jpg size_local : 81791 size in s3 : 81791 create time local : 2023-03-03 12:26:15 create time in s3 : 2023-02-22 12:54:30 Precision_Recall_cartonnette.jpg already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/Precision_Recall_environnement.jpg size_local : 59936 size in s3 : 59936 create time local : 2023-03-03 12:26:15 create time in s3 : 2023-02-22 12:54:17 Precision_Recall_environnement.jpg already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/Precision_Recall_kraft.jpg size_local : 80280 size in s3 : 80280 create time local : 2023-03-03 12:26:15 create time in s3 : 2023-02-22 12:54:25 Precision_Recall_kraft.jpg already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/Precision_Recall_metal.jpg size_local : 77736 size in s3 : 77736 create time local : 2023-03-03 12:26:15 create time in s3 : 2023-02-22 12:54:31 Precision_Recall_metal.jpg already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/Precision_Recall_papier.jpg size_local : 69041 size in s3 : 69041 create time local : 2023-03-03 12:26:15 create time in s3 : 2023-02-22 12:54:31 Precision_Recall_papier.jpg already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/Precision_Recall_plastique.jpg size_local : 70770 size in s3 : 70770 create time local : 2023-03-03 12:26:15 create time in s3 : 2023-02-22 12:54:17 Precision_Recall_plastique.jpg already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/Result_Summary.txt size_local : 1092 size in s3 : 1092 create time local : 2023-03-03 12:26:15 create time in s3 : 2023-02-22 12:54:31 Result_Summary.txt already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-03-03 12:26:15 create time in s3 : 2023-02-22 12:54:31 checkpoint already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/model_checkpoint.ckpt.data-00000-of-00002 size_local : 429801 size in s3 : 429801 create time local : 2023-03-03 12:26:16 create time in s3 : 2023-02-22 12:54:30 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/model_checkpoint.ckpt.data-00001-of-00002 size_local : 141033188 size in s3 : 141033188 create time local : 2023-03-03 12:26:17 create time in s3 : 2023-02-22 12:54:25 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/model_checkpoint.ckpt.index size_local : 85618 size in s3 : 85618 create time local : 2023-03-03 12:26:17 create time in s3 : 2023-02-22 12:54:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/learn_papier_mixte_centre_tf_b4_220223_3/model_weights.h5 size_local : 71289448 size in s3 : 71289448 create time local : 2023-03-03 12:26:18 create time in s3 : 2023-02-22 12:54:17 model_weights.h5 already exist and didn't need to update desc size : 1792 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 1792) 17673816 _________________________________________________________________ learn_papier_mixte_centre_tf (None, 10) 17930 ================================================================= Total params: 17,691,746 Trainable params: 17,930 Non-trainable params: 17,673,816 _________________________________________________________________ Loading Weights... time used to create the model : 16.98819875717163 time used to load_weights : 0.30550265312194824 0it [00:00, ?it/s] 69it [00:00, 1180.88it/s]2025-09-26 11:22:59.571158: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-26 11:22:59.578151: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 temp/1758878412_55108_1385110088_c2c44a80a82422996eb48d1a5d6bf78a_rle_crop_3969766785_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3969766798_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3969766796_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3969766794_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3969766801_0.png temp/1758878412_55108_1385110083_3418aa80c4e5ac0af00e113a35858cad_rle_crop_3969766817_0.png temp/1758878412_55108_1385110083_3418aa80c4e5ac0af00e113a35858cad_rle_crop_3969766812_0.png temp/1758878412_55108_1385110083_3418aa80c4e5ac0af00e113a35858cad_rle_crop_3976635422_0.png temp/1758878412_55108_1385110080_c523720d61dc6ec43d74417a246ab5b1_rle_crop_3969766825_0.png temp/1758878412_55108_1385110080_c523720d61dc6ec43d74417a246ab5b1_rle_crop_3969766820_0.png temp/1758878412_55108_1385106321_0d00a498e174ded8b00480e89774ce91_rle_crop_3969766863_0.png temp/1758878412_55108_1385106321_0d00a498e174ded8b00480e89774ce91_rle_crop_3969766861_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3969766809_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3969766810_0.png temp/1758878412_55108_1385110080_c523720d61dc6ec43d74417a246ab5b1_rle_crop_3969766828_0.png temp/1758878412_55108_1385106334_a523e0211eed610730ad60c18c07340f_rle_crop_3969766832_0.png temp/1758878412_55108_1385106334_a523e0211eed610730ad60c18c07340f_rle_crop_3969766830_0.png temp/1758878412_55108_1385106334_a523e0211eed610730ad60c18c07340f_rle_crop_3969766840_0.png temp/1758878412_55108_1385106321_0d00a498e174ded8b00480e89774ce91_rle_crop_3969766869_0.png temp/1758878412_55108_1385106331_30f210a2c4bc4024eae08a180b680752_rle_crop_3969766860_0.png temp/1758878412_55108_1385106331_30f210a2c4bc4024eae08a180b680752_rle_crop_3969766855_0.png temp/1758878412_55108_1385106331_30f210a2c4bc4024eae08a180b680752_rle_crop_3969766856_0.png temp/1758878412_55108_1385110088_c2c44a80a82422996eb48d1a5d6bf78a_rle_crop_3969766786_0.png temp/1758878412_55108_1385110088_c2c44a80a82422996eb48d1a5d6bf78a_rle_crop_3969766789_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3969766797_0.png temp/1758878412_55108_1385110083_3418aa80c4e5ac0af00e113a35858cad_rle_crop_3969766813_0.png temp/1758878412_55108_1385110083_3418aa80c4e5ac0af00e113a35858cad_rle_crop_3969766816_0.png temp/1758878412_55108_1385110083_3418aa80c4e5ac0af00e113a35858cad_rle_crop_3969766819_0.png temp/1758878412_55108_1385110083_3418aa80c4e5ac0af00e113a35858cad_rle_crop_3969766815_0.png temp/1758878412_55108_1385110080_c523720d61dc6ec43d74417a246ab5b1_rle_crop_3969766821_0.png temp/1758878412_55108_1385106334_a523e0211eed610730ad60c18c07340f_rle_crop_3969766834_0.png temp/1758878412_55108_1385106332_a040d9db68531ad6e9aa60f81be6edd5_rle_crop_3969766849_0.png temp/1758878412_55108_1385106331_30f210a2c4bc4024eae08a180b680752_rle_crop_3969766857_0.png temp/1758878412_55108_1385110088_c2c44a80a82422996eb48d1a5d6bf78a_rle_crop_3969766791_0.png temp/1758878412_55108_1385110088_c2c44a80a82422996eb48d1a5d6bf78a_rle_crop_3969766788_0.png temp/1758878412_55108_1385110088_c2c44a80a82422996eb48d1a5d6bf78a_rle_crop_3969766790_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3969766799_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3969766802_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3969766806_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3969766795_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3969766804_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3969766805_0.png temp/1758878412_55108_1385110083_3418aa80c4e5ac0af00e113a35858cad_rle_crop_3969766811_0.png temp/1758878412_55108_1385110080_c523720d61dc6ec43d74417a246ab5b1_rle_crop_3969766822_0.png temp/1758878412_55108_1385110080_c523720d61dc6ec43d74417a246ab5b1_rle_crop_3969766823_0.png temp/1758878412_55108_1385106334_a523e0211eed610730ad60c18c07340f_rle_crop_3969766831_0.png temp/1758878412_55108_1385106334_a523e0211eed610730ad60c18c07340f_rle_crop_3969766829_0.png temp/1758878412_55108_1385106334_a523e0211eed610730ad60c18c07340f_rle_crop_3969766833_0.png temp/1758878412_55108_1385106332_a040d9db68531ad6e9aa60f81be6edd5_rle_crop_3969766847_0.png temp/1758878412_55108_1385106332_a040d9db68531ad6e9aa60f81be6edd5_rle_crop_3969766845_0.png temp/1758878412_55108_1385106332_a040d9db68531ad6e9aa60f81be6edd5_rle_crop_3969766844_0.png temp/1758878412_55108_1385106332_a040d9db68531ad6e9aa60f81be6edd5_rle_crop_3969766842_0.png temp/1758878412_55108_1385106332_a040d9db68531ad6e9aa60f81be6edd5_rle_crop_3976635424_0.png temp/1758878412_55108_1385106331_30f210a2c4bc4024eae08a180b680752_rle_crop_3969766854_0.png temp/1758878412_55108_1385106331_30f210a2c4bc4024eae08a180b680752_rle_crop_3969766852_0.png temp/1758878412_55108_1385106331_30f210a2c4bc4024eae08a180b680752_rle_crop_3969766853_0.png temp/1758878412_55108_1385106321_0d00a498e174ded8b00480e89774ce91_rle_crop_3969766865_0.png temp/1758878412_55108_1385106321_0d00a498e174ded8b00480e89774ce91_rle_crop_3969766862_0.png temp/1758878412_55108_1385106321_0d00a498e174ded8b00480e89774ce91_rle_crop_3969766867_0.png temp/1758878412_55108_1385106321_0d00a498e174ded8b00480e89774ce91_rle_crop_3969766868_0.png temp/1758878412_55108_1385106321_0d00a498e174ded8b00480e89774ce91_rle_crop_3969766866_0.png temp/1758878412_55108_1385110088_c2c44a80a82422996eb48d1a5d6bf78a_rle_crop_3969766793_0.png temp/1758878412_55108_1385110088_c2c44a80a82422996eb48d1a5d6bf78a_rle_crop_3969766787_0.png temp/1758878412_55108_1385110084_869ab5faeb4f22569bf5e123bfcb286a_rle_crop_3976635421_0.png temp/1758878412_55108_1385106334_a523e0211eed610730ad60c18c07340f_rle_crop_3969766836_0.png temp/1758878412_55108_1385106334_a523e0211eed610730ad60c18c07340f_rle_crop_3969766835_0.png temp/1758878412_55108_1385106334_a523e0211eed610730ad60c18c07340f_rle_crop_3969766837_0.png temp/1758878412_55108_1385106334_a523e0211eed610730ad60c18c07340f_rle_crop_3969766841_0.png temp/1758878412_55108_1385106321_0d00a498e174ded8b00480e89774ce91_rle_crop_3969766872_0.png Found 69 images belonging to 1 classes. begin to do the prediction : time used to do the prediction : 5.081537246704102 (69,) (69, 10) (69, 1792) shape of features : (69, 1792) shape of new features : (1, 69, 1792) save descriptor for thcl : 3501 time to traite the descriptors : 0.3674588203430176 storage_type for insertDescriptorsMulti : 3 Missing photo l117 : -3969766785 Missing photo l117 : -3969766798 Missing photo l117 : -3969766796 Missing photo l117 : -3969766794 Missing photo l117 : -3969766801 Missing photo l117 : -3969766817 Missing photo l117 : -3969766812 Missing photo l117 : -3976635422 Missing photo l117 : -3969766825 Missing photo l117 : -3969766820 Missing photo l117 : -3969766863 Missing photo l117 : -3969766861 Missing photo l117 : -3969766809 Missing photo l117 : -3969766810 Missing photo l117 : -3969766828 Missing photo l117 : -3969766832 Missing photo l117 : -3969766830 Missing photo l117 : -3969766840 Missing photo l117 : -3969766869 Missing photo l117 : -3969766860 Missing photo l117 : -3969766855 Missing photo l117 : -3969766856 Missing photo l117 : -3969766786 Missing photo l117 : -3969766789 Missing photo l117 : -3969766797 Missing photo l117 : -3969766813 Missing photo l117 : -3969766816 Missing photo l117 : -3969766819 Missing photo l117 : -3969766815 Missing photo l117 : -3969766821 Missing photo l117 : -3969766834 Missing photo l117 : -3969766849 Missing photo l117 : -3969766857 Missing photo l117 : -3969766791 Missing photo l117 : -3969766788 Missing photo l117 : -3969766790 Missing photo l117 : -3969766799 Missing photo l117 : -3969766802 Missing photo l117 : -3969766806 Missing photo l117 : -3969766795 Missing photo l117 : -3969766804 Missing photo l117 : -3969766805 Missing photo l117 : -3969766811 Missing photo l117 : -3969766822 Missing photo l117 : -3969766823 Missing photo l117 : -3969766831 Missing photo l117 : -3969766829 Missing photo l117 : -3969766833 Missing photo l117 : -3969766847 Missing photo l117 : -3969766845 Missing photo l117 : -3969766844 Missing photo l117 : -3969766842 Missing photo l117 : -3976635424 Missing photo l117 : -3969766854 Missing photo l117 : -3969766852 Missing photo l117 : -3969766853 Missing photo l117 : -3969766865 Missing photo l117 : -3969766862 Missing photo l117 : -3969766867 Missing photo l117 : -3969766868 Missing photo l117 : -3969766866 Missing photo l117 : -3969766793 Missing photo l117 : -3969766787 Missing photo l117 : -3976635421 Missing photo l117 : -3969766836 Missing photo l117 : -3969766835 Missing photo l117 : -3969766837 Missing photo l117 : -3969766841 Missing photo l117 : -3969766872 To insert : -3969766785 Missing photo l134 : -3969766785 To insert : -3969766798 Missing photo l134 : -3969766798 To insert : -3969766796 Missing photo l134 : -3969766796 To insert : -3969766794 Missing photo l134 : -3969766794 To insert : -3969766801 Missing photo l134 : -3969766801 To insert : -3969766817 Missing photo l134 : -3969766817 To insert : -3969766812 Missing photo l134 : -3969766812 To insert : -3976635422 Missing photo l134 : -3976635422 To insert : -3969766825 Missing photo l134 : -3969766825 To insert : -3969766820 Missing photo l134 : -3969766820 To insert : -3969766863 Missing photo l134 : -3969766863 To insert : -3969766861 Missing photo l134 : -3969766861 To insert : -3969766809 Missing photo l134 : -3969766809 To insert : -3969766810 Missing photo l134 : -3969766810 To insert : -3969766828 Missing photo l134 : -3969766828 To insert : -3969766832 Missing photo l134 : -3969766832 To insert : -3969766830 Missing photo l134 : -3969766830 To insert : -3969766840 Missing photo l134 : -3969766840 To insert : -3969766869 Missing photo l134 : -3969766869 To insert : -3969766860 Missing photo l134 : -3969766860 To insert : -3969766855 Missing photo l134 : -3969766855 To insert : -3969766856 Missing photo l134 : -3969766856 To insert : -3969766786 Missing photo l134 : -3969766786 To insert : -3969766789 Missing photo l134 : -3969766789 To insert : -3969766797 Missing photo l134 : -3969766797 To insert : -3969766813 Missing photo l134 : -3969766813 To insert : -3969766816 Missing photo l134 : -3969766816 To insert : -3969766819 Missing photo l134 : -3969766819 To insert : -3969766815 Missing photo l134 : -3969766815 To insert : -3969766821 Missing photo l134 : -3969766821 To insert : -3969766834 Missing photo l134 : -3969766834 To insert : -3969766849 Missing photo l134 : -3969766849 To insert : -3969766857 Missing photo l134 : -3969766857 To insert : -3969766791 Missing photo l134 : -3969766791 To insert : -3969766788 Missing photo l134 : -3969766788 To insert : -3969766790 Missing photo l134 : -3969766790 To insert : -3969766799 Missing photo l134 : -3969766799 To insert : -3969766802 Missing photo l134 : -3969766802 To insert : -3969766806 Missing photo l134 : -3969766806 To insert : -3969766795 Missing photo l134 : -3969766795 To insert : -3969766804 Missing photo l134 : -3969766804 To insert : -3969766805 Missing photo l134 : -3969766805 To insert : -3969766811 Missing photo l134 : -3969766811 To insert : -3969766822 Missing photo l134 : -3969766822 To insert : -3969766823 Missing photo l134 : -3969766823 To insert : -3969766831 Missing photo l134 : -3969766831 To insert : -3969766829 Missing photo l134 : -3969766829 To insert : -3969766833 Missing photo l134 : -3969766833 To insert : -3969766847 Missing photo l134 : -3969766847 To insert : -3969766845 Missing photo l134 : -3969766845 To insert : -3969766844 Missing photo l134 : -3969766844 To insert : -3969766842 Missing photo l134 : -3969766842 To insert : -3976635424 Missing photo l134 : -3976635424 To insert : -3969766854 Missing photo l134 : -3969766854 To insert : -3969766852 Missing photo l134 : -3969766852 To insert : -3969766853 Missing photo l134 : -3969766853 To insert : -3969766865 Missing photo l134 : -3969766865 To insert : -3969766862 Missing photo l134 : -3969766862 To insert : -3969766867 Missing photo l134 : -3969766867 To insert : -3969766868 Missing photo l134 : -3969766868 To insert : -3969766866 Missing photo l134 : -3969766866 To insert : -3969766793 Missing photo l134 : -3969766793 To insert : -3969766787 Missing photo l134 : -3969766787 To insert : -3976635421 Missing photo l134 : -3976635421 To insert : -3969766836 Missing photo l134 : -3969766836 To insert : -3969766835 Missing photo l134 : -3969766835 To insert : -3969766837 Missing photo l134 : -3969766837 To insert : -3969766841 Missing photo l134 : -3969766841 To insert : -3969766872 Missing photo l134 : -3969766872 time to insert the descriptors : 18.361189603805542 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : tfhub_classification2 we use saveGeneral [1385110088, 1385110084, 1385110083, 1385110080, 1385106334, 1385106332, 1385106331, 1385106321] Looping around the photos to save general results len do output : 69 /-3969766785Didn't retrieve data . /-3969766798Didn't retrieve data . /-3969766796Didn't retrieve data . /-3969766794Didn't retrieve data . /-3969766801Didn't retrieve data . /-3969766817Didn't retrieve data . /-3969766812Didn't retrieve data . /-3976635422Didn't retrieve data . /-3969766825Didn't retrieve data . /-3969766820Didn't retrieve data . /-3969766863Didn't retrieve data . /-3969766861Didn't retrieve data . /-3969766809Didn't retrieve data . /-3969766810Didn't retrieve data . /-3969766828Didn't retrieve data . /-3969766832Didn't retrieve data . /-3969766830Didn't retrieve data . /-3969766840Didn't retrieve data . /-3969766869Didn't retrieve data . /-3969766860Didn't retrieve data . /-3969766855Didn't retrieve data . /-3969766856Didn't retrieve data . /-3969766786Didn't retrieve data . /-3969766789Didn't retrieve data . /-3969766797Didn't retrieve data . /-3969766813Didn't retrieve data . /-3969766816Didn't retrieve data . /-3969766819Didn't retrieve data . /-3969766815Didn't retrieve data . /-3969766821Didn't retrieve data . /-3969766834Didn't retrieve data . /-3969766849Didn't retrieve data . /-3969766857Didn't retrieve data . /-3969766791Didn't retrieve data . /-3969766788Didn't retrieve data . /-3969766790Didn't retrieve data . /-3969766799Didn't retrieve data . /-3969766802Didn't retrieve data . /-3969766806Didn't retrieve data . /-3969766795Didn't retrieve data . /-3969766804Didn't retrieve data . /-3969766805Didn't retrieve data . /-3969766811Didn't retrieve data . /-3969766822Didn't retrieve data . /-3969766823Didn't retrieve data . /-3969766831Didn't retrieve data . /-3969766829Didn't retrieve data . /-3969766833Didn't retrieve data . /-3969766847Didn't retrieve data . /-3969766845Didn't retrieve data . /-3969766844Didn't retrieve data . /-3969766842Didn't retrieve data . /-3976635424Didn't retrieve data . /-3969766854Didn't retrieve data . /-3969766852Didn't retrieve data . /-3969766853Didn't retrieve data . /-3969766865Didn't retrieve data . /-3969766862Didn't retrieve data . /-3969766867Didn't retrieve data . /-3969766868Didn't retrieve data . /-3969766866Didn't retrieve data . /-3969766793Didn't retrieve data . /-3969766787Didn't retrieve data . /-3976635421Didn't retrieve data . /-3969766836Didn't retrieve data . /-3969766835Didn't retrieve data . /-3969766837Didn't retrieve data . /-3969766841Didn't retrieve data . /-3969766872Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110088', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110084', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110083', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110080', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106334', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106332', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106331', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106321', None, None, None, None, None, '3784106') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 77 time used for this insertion : 0.022815465927124023 save_final save missing photos in datou_result : time spend for datou_step_exec : 48.299683570861816 time spend to save output : 0.02384805679321289 total time spend for step 6 : 48.32353162765503 step7:merge_mask_thcl_custom Fri Sep 26 11:23:23 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 94 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1385110088, 1385110084, 1385110083, 1385110080, 1385106334, 1385106332, 1385106331, 1385106321] Looping around the photos to save general results len do output : 8 /1385110088Didn't retrieve data .Didn't retrieve data . /1385110084Didn't retrieve data .Didn't retrieve data . /1385110083Didn't retrieve data .Didn't retrieve data . /1385110080Didn't retrieve data .Didn't retrieve data . /1385106334Didn't retrieve data .Didn't retrieve data . /1385106332Didn't retrieve data .Didn't retrieve data . /1385106331Didn't retrieve data .Didn't retrieve data . /1385106321Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110088', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110084', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110083', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110080', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106334', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106332', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106331', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106321', None, None, None, None, None, '3784106') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 24 time used for this insertion : 0.012442588806152344 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.06512880325317383 time spend to save output : 0.012825965881347656 total time spend for step 7 : 0.07795476913452148 step8:rle_unique_nms_with_priority Fri Sep 26 11:23:23 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 10 nb_hashtags : 6 time to prepare the origin masks : 10.393547534942627 time for calcul the mask position with numpy : 0.49704456329345703 nb_pixel_total : 10405085 time to create 1 rle with new method : 0.6083581447601318 time for calcul the mask position with numpy : 0.07056975364685059 nb_pixel_total : 353950 time to create 1 rle with new method : 0.6817386150360107 time for calcul the mask position with numpy : 0.05992841720581055 nb_pixel_total : 82072 time to create 1 rle with old method : 0.0938425064086914 time for calcul the mask position with numpy : 0.06099987030029297 nb_pixel_total : 231025 time to create 1 rle with new method : 0.7089385986328125 time for calcul the mask position with numpy : 0.05183076858520508 nb_pixel_total : 2990 time to create 1 rle with old method : 0.0037822723388671875 time for calcul the mask position with numpy : 0.05672717094421387 nb_pixel_total : 664532 time to create 1 rle with new method : 0.7091226577758789 time for calcul the mask position with numpy : 0.05689668655395508 nb_pixel_total : 48124 time to create 1 rle with old method : 0.052292585372924805 time for calcul the mask position with numpy : 0.056931495666503906 nb_pixel_total : 135012 time to create 1 rle with old method : 0.14544272422790527 time for calcul the mask position with numpy : 0.05565190315246582 nb_pixel_total : 185334 time to create 1 rle with new method : 0.6268081665039062 time for calcul the mask position with numpy : 0.05773162841796875 nb_pixel_total : 103397 time to create 1 rle with old method : 0.11655020713806152 time for calcul the mask position with numpy : 0.05877947807312012 nb_pixel_total : 118719 time to create 1 rle with old method : 0.16282963752746582 create new chi : 5.199103593826294 time to delete rle : 0.016268491744995117 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 16463 TO DO : save crop sub photo not yet done ! save time : 0.8707864284515381 nb_obj : 15 nb_hashtags : 5 time to prepare the origin masks : 13.67289686203003 time for calcul the mask position with numpy : 0.4119093418121338 nb_pixel_total : 10897582 time to create 1 rle with new method : 0.6139061450958252 time for calcul the mask position with numpy : 0.06231403350830078 nb_pixel_total : 173892 time to create 1 rle with new method : 0.6250503063201904 time for calcul the mask position with numpy : 0.058141469955444336 nb_pixel_total : 13047 time to create 1 rle with old method : 0.014592885971069336 time for calcul the mask position with numpy : 0.04142570495605469 nb_pixel_total : 486862 time to create 1 rle with new method : 0.57696533203125 time for calcul the mask position with numpy : 0.03751635551452637 nb_pixel_total : 70873 time to create 1 rle with old method : 0.08543944358825684 time for calcul the mask position with numpy : 0.04271674156188965 nb_pixel_total : 13722 time to create 1 rle with old method : 0.0182497501373291 time for calcul the mask position with numpy : 0.04812169075012207 nb_pixel_total : 43250 time to create 1 rle with old method : 0.0587465763092041 time for calcul the mask position with numpy : 0.03769707679748535 nb_pixel_total : 883 time to create 1 rle with old method : 0.001298666000366211 time for calcul the mask position with numpy : 0.036307334899902344 nb_pixel_total : 417 time to create 1 rle with old method : 0.0005972385406494141 time for calcul the mask position with numpy : 0.039836883544921875 nb_pixel_total : 193408 time to create 1 rle with new method : 0.6709680557250977 time for calcul the mask position with numpy : 0.05902504920959473 nb_pixel_total : 40833 time to create 1 rle with old method : 0.049550533294677734 time for calcul the mask position with numpy : 0.07146835327148438 nb_pixel_total : 25340 time to create 1 rle with old method : 0.027782678604125977 time for calcul the mask position with numpy : 0.04902458190917969 nb_pixel_total : 268879 time to create 1 rle with new method : 0.5362114906311035 time for calcul the mask position with numpy : 0.05853080749511719 nb_pixel_total : 4045 time to create 1 rle with old method : 0.004297733306884766 time for calcul the mask position with numpy : 0.049836158752441406 nb_pixel_total : 13638 time to create 1 rle with old method : 0.015386581420898438 time for calcul the mask position with numpy : 0.05451178550720215 nb_pixel_total : 83569 time to create 1 rle with old method : 0.09078621864318848 create new chi : 4.755897045135498 time to delete rle : 0.0013165473937988281 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 12396 TO DO : save crop sub photo not yet done ! save time : 0.6515676975250244 nb_obj : 9 nb_hashtags : 6 time to prepare the origin masks : 11.28783369064331 time for calcul the mask position with numpy : 0.5296194553375244 nb_pixel_total : 10622804 time to create 1 rle with new method : 0.536510705947876 time for calcul the mask position with numpy : 0.061215877532958984 nb_pixel_total : 405046 time to create 1 rle with new method : 0.5243957042694092 time for calcul the mask position with numpy : 0.05606651306152344 nb_pixel_total : 21107 time to create 1 rle with old method : 0.0228574275970459 time for calcul the mask position with numpy : 0.05636477470397949 nb_pixel_total : 25908 time to create 1 rle with old method : 0.028191804885864258 time for calcul the mask position with numpy : 0.0507051944732666 nb_pixel_total : 150205 time to create 1 rle with new method : 0.6205358505249023 time for calcul the mask position with numpy : 0.038542985916137695 nb_pixel_total : 423063 time to create 1 rle with new method : 0.56339430809021 time for calcul the mask position with numpy : 0.040708303451538086 nb_pixel_total : 255119 time to create 1 rle with new method : 0.633887767791748 time for calcul the mask position with numpy : 0.05776238441467285 nb_pixel_total : 319897 time to create 1 rle with new method : 0.5928125381469727 time for calcul the mask position with numpy : 0.05804562568664551 nb_pixel_total : 58926 time to create 1 rle with old method : 0.06745028495788574 time for calcul the mask position with numpy : 0.06618094444274902 nb_pixel_total : 48165 time to create 1 rle with old method : 0.06342911720275879 create new chi : 4.912071704864502 time to delete rle : 0.002176523208618164 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 13294 TO DO : save crop sub photo not yet done ! save time : 0.7231495380401611 nb_obj : 8 nb_hashtags : 3 time to prepare the origin masks : 9.0481595993042 time for calcul the mask position with numpy : 0.5066025257110596 nb_pixel_total : 10531320 time to create 1 rle with new method : 0.577531099319458 time for calcul the mask position with numpy : 0.039228200912475586 nb_pixel_total : 130472 time to create 1 rle with old method : 0.15534353256225586 time for calcul the mask position with numpy : 0.0372159481048584 nb_pixel_total : 49936 time to create 1 rle with old method : 0.05583953857421875 time for calcul the mask position with numpy : 0.03660011291503906 nb_pixel_total : 11293 time to create 1 rle with old method : 0.012551069259643555 time for calcul the mask position with numpy : 0.04323220252990723 nb_pixel_total : 632278 time to create 1 rle with new method : 0.6444058418273926 time for calcul the mask position with numpy : 0.03806638717651367 nb_pixel_total : 3998 time to create 1 rle with old method : 0.005231142044067383 time for calcul the mask position with numpy : 0.045545339584350586 nb_pixel_total : 861103 time to create 1 rle with new method : 0.6206858158111572 time for calcul the mask position with numpy : 0.040287017822265625 nb_pixel_total : 65641 time to create 1 rle with old method : 0.07477259635925293 time for calcul the mask position with numpy : 0.04087686538696289 nb_pixel_total : 44199 time to create 1 rle with old method : 0.054110050201416016 create new chi : 3.165708541870117 time to delete rle : 0.001737833023071289 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 14114 TO DO : save crop sub photo not yet done ! save time : 0.7267475128173828 nb_obj : 11 nb_hashtags : 7 time to prepare the origin masks : 12.74786376953125 time for calcul the mask position with numpy : 0.8110120296478271 nb_pixel_total : 11061135 time to create 1 rle with new method : 0.5465433597564697 time for calcul the mask position with numpy : 0.06043267250061035 nb_pixel_total : 66648 time to create 1 rle with old method : 0.0720968246459961 time for calcul the mask position with numpy : 0.0596613883972168 nb_pixel_total : 4080 time to create 1 rle with old method : 0.004729270935058594 time for calcul the mask position with numpy : 0.03479337692260742 nb_pixel_total : 59433 time to create 1 rle with old method : 0.06577467918395996 time for calcul the mask position with numpy : 0.03827381134033203 nb_pixel_total : 260634 time to create 1 rle with new method : 0.6209309101104736 time for calcul the mask position with numpy : 0.05067944526672363 nb_pixel_total : 225949 time to create 1 rle with new method : 0.7988426685333252 time for calcul the mask position with numpy : 0.041562795639038086 nb_pixel_total : 107234 time to create 1 rle with old method : 0.11605024337768555 time for calcul the mask position with numpy : 0.03540778160095215 nb_pixel_total : 142870 time to create 1 rle with old method : 0.15776920318603516 time for calcul the mask position with numpy : 0.060500383377075195 nb_pixel_total : 40935 time to create 1 rle with old method : 0.044460296630859375 time for calcul the mask position with numpy : 0.05786705017089844 nb_pixel_total : 20142 time to create 1 rle with old method : 0.023014545440673828 time for calcul the mask position with numpy : 0.06177520751953125 nb_pixel_total : 337388 time to create 1 rle with new method : 0.7203576564788818 time for calcul the mask position with numpy : 0.06337738037109375 nb_pixel_total : 3792 time to create 1 rle with old method : 0.004360198974609375 create new chi : 4.745037794113159 time to delete rle : 0.0012636184692382812 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 12028 TO DO : save crop sub photo not yet done ! save time : 0.6369171142578125 nb_obj : 10 nb_hashtags : 2 time to prepare the origin masks : 9.420468807220459 time for calcul the mask position with numpy : 0.6490116119384766 nb_pixel_total : 9690793 time to create 1 rle with new method : 0.7599537372589111 time for calcul the mask position with numpy : 0.03610658645629883 nb_pixel_total : 24680 time to create 1 rle with old method : 0.029238224029541016 time for calcul the mask position with numpy : 0.04038810729980469 nb_pixel_total : 14442 time to create 1 rle with old method : 0.016153573989868164 time for calcul the mask position with numpy : 0.05438399314880371 nb_pixel_total : 1670436 time to create 1 rle with new method : 0.8548293113708496 time for calcul the mask position with numpy : 0.05739998817443848 nb_pixel_total : 707369 time to create 1 rle with new method : 0.9260938167572021 time for calcul the mask position with numpy : 0.03576302528381348 nb_pixel_total : 15028 time to create 1 rle with old method : 0.01624321937561035 time for calcul the mask position with numpy : 0.03603053092956543 nb_pixel_total : 7952 time to create 1 rle with old method : 0.00851130485534668 time for calcul the mask position with numpy : 0.0355074405670166 nb_pixel_total : 36923 time to create 1 rle with old method : 0.04098320007324219 time for calcul the mask position with numpy : 0.03637099266052246 nb_pixel_total : 11637 time to create 1 rle with old method : 0.013415098190307617 time for calcul the mask position with numpy : 0.03720831871032715 nb_pixel_total : 147046 time to create 1 rle with old method : 0.15885400772094727 time for calcul the mask position with numpy : 0.0366663932800293 nb_pixel_total : 3934 time to create 1 rle with old method : 0.00435948371887207 create new chi : 4.027042388916016 time to delete rle : 0.0015690326690673828 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 16686 TO DO : save crop sub photo not yet done ! save time : 0.8504824638366699 nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 9.74950885772705 time for calcul the mask position with numpy : 0.8783612251281738 nb_pixel_total : 11874167 time to create 1 rle with new method : 0.9312365055084229 time for calcul the mask position with numpy : 0.06058001518249512 nb_pixel_total : 74234 time to create 1 rle with old method : 0.08069181442260742 time for calcul the mask position with numpy : 0.05872654914855957 nb_pixel_total : 92206 time to create 1 rle with old method : 0.10108304023742676 time for calcul the mask position with numpy : 0.043454885482788086 nb_pixel_total : 33827 time to create 1 rle with old method : 0.04643678665161133 time for calcul the mask position with numpy : 0.042067766189575195 nb_pixel_total : 93034 time to create 1 rle with old method : 0.1012880802154541 time for calcul the mask position with numpy : 0.0381467342376709 nb_pixel_total : 119411 time to create 1 rle with old method : 0.12826275825500488 time for calcul the mask position with numpy : 0.035338640213012695 nb_pixel_total : 30393 time to create 1 rle with old method : 0.03322625160217285 time for calcul the mask position with numpy : 0.0433652400970459 nb_pixel_total : 12968 time to create 1 rle with old method : 0.016233444213867188 create new chi : 2.712167978286743 time to delete rle : 0.0012004375457763672 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 9518 TO DO : save crop sub photo not yet done ! save time : 0.5149884223937988 nb_obj : 10 nb_hashtags : 4 time to prepare the origin masks : 10.37791132926941 time for calcul the mask position with numpy : 0.8358752727508545 nb_pixel_total : 11766720 time to create 1 rle with new method : 0.5686278343200684 time for calcul the mask position with numpy : 0.036051273345947266 nb_pixel_total : 21447 time to create 1 rle with old method : 0.023128747940063477 time for calcul the mask position with numpy : 0.036321401596069336 nb_pixel_total : 30727 time to create 1 rle with old method : 0.03304457664489746 time for calcul the mask position with numpy : 0.03605508804321289 nb_pixel_total : 62565 time to create 1 rle with old method : 0.07064151763916016 time for calcul the mask position with numpy : 0.03986549377441406 nb_pixel_total : 72027 time to create 1 rle with old method : 0.08349919319152832 time for calcul the mask position with numpy : 0.03572201728820801 nb_pixel_total : 23496 time to create 1 rle with old method : 0.02586197853088379 time for calcul the mask position with numpy : 0.03667426109313965 nb_pixel_total : 71213 time to create 1 rle with old method : 0.07836699485778809 time for calcul the mask position with numpy : 0.03692054748535156 nb_pixel_total : 22324 time to create 1 rle with old method : 0.024571657180786133 time for calcul the mask position with numpy : 0.036118507385253906 nb_pixel_total : 215382 time to create 1 rle with new method : 0.7535769939422607 time for calcul the mask position with numpy : 0.038710594177246094 nb_pixel_total : 11884 time to create 1 rle with old method : 0.013428926467895508 time for calcul the mask position with numpy : 0.04003500938415527 nb_pixel_total : 32455 time to create 1 rle with old method : 0.03580808639526367 create new chi : 3.0334763526916504 time to delete rle : 0.001073598861694336 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 9478 TO DO : save crop sub photo not yet done ! save time : 0.4931039810180664 map_output_result : {1385110088: (0.0, 'Should be the crop_list due to order', 0.0), 1385110084: (0.0, 'Should be the crop_list due to order', 0.0), 1385110083: (0.0, 'Should be the crop_list due to order', 0.0), 1385110080: (0.0, 'Should be the crop_list due to order', 0.0), 1385106334: (0.0, 'Should be the crop_list due to order', 0.0), 1385106332: (0.0, 'Should be the crop_list due to order', 0.0), 1385106331: (0.0, 'Should be the crop_list due to order', 0.0), 1385106321: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1385110088, 1385110084, 1385110083, 1385110080, 1385106334, 1385106332, 1385106331, 1385106321] Looping around the photos to save general results len do output : 8 /1385110088.Didn't retrieve data . /1385110084.Didn't retrieve data . /1385110083.Didn't retrieve data . /1385110080.Didn't retrieve data . /1385106334.Didn't retrieve data . /1385106332.Didn't retrieve data . /1385106331.Didn't retrieve data . /1385106321.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110088', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110084', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110083', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110080', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106334', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106332', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106331', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106321', None, None, None, None, None, '3784106') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 24 time used for this insertion : 0.013657569885253906 save_final save missing photos in datou_result : time spend for datou_step_exec : 127.1195638179779 time spend to save output : 0.014089822769165039 total time spend for step 8 : 127.13365364074707 step9:crop_condition Fri Sep 26 11:25:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 8 ! batch 1 Loaded 92 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1758878738_55108 batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first Unexecpected behavior in 07/2025 that can be generalized l287 : type_extension .jpg This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 3.7886693477630615 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1758878744_55108 batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first Unexecpected behavior in 07/2025 that can be generalized l287 : type_extension .jpg This is a hack ! we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5835158824920654 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1758878748_55108 batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first Unexecpected behavior in 07/2025 that can be generalized l287 : type_extension .jpg This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first Unexecpected behavior in 07/2025 that can be generalized l287 : type_extension .jpg This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.0786049365997314 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 20 About to insert : list_path_to_insert length 20 new photo from crops ! About to upload 20 photos upload in portfolio : 4869462 init cache_photo without model_param we have 20 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1758878757_55108 batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first Unexecpected behavior in 07/2025 that can be generalized l287 : type_extension .jpg This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first Unexecpected behavior in 07/2025 that can be generalized l287 : type_extension .jpg This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first Unexecpected behavior in 07/2025 that can be generalized l287 : type_extension .jpg This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first Unexecpected behavior in 07/2025 that can be generalized l287 : type_extension .jpg This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! we have uploaded 20 photos in the portfolio 4869462 time of upload the photos Elapsed time : 5.414744138717651 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1758878766_55108 batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.765399694442749 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1758878769_55108 batch_size : 0, verbose : False, strat_bulk_insert : ignore_different_from_first This is a hack ! we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6655654907226562 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1385110088, 1385110084, 1385110083, 1385110080, 1385106334, 1385106332, 1385106331, 1385106321] Looping around the photos to save general results len do output : 34 /1386331217Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331218Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331219Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331220Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331221Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331222Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331223Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331224Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331225Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331229Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331230Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331231Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331265Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331266Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331268Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331269Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331270Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331271Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331273Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331274Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331275Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331277Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331278Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331279Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331280Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331282Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331283Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331284Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331286Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331287Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331288Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331290Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331319Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1386331342Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110088', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110084', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110083', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110080', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106334', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106332', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106331', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106321', None, None, None, None, None, '3784106') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 110 time used for this insertion : 0.026102781295776367 save_final save missing photos in datou_result : time spend for datou_step_exec : 39.62291669845581 time spend to save output : 0.027631759643554688 total time spend for step 9 : 39.650548458099365 step10:ventilate_hashtags_in_portfolio Fri Sep 26 11:26:10 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 27267728 get user id for portfolio 27267728 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=27267728 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('mal_croppe','flou','Carton_brun','papier','environnement','plastique','cartonnette','Carton_gris','Teint_Dans_La_Masse','autre_refus','metal','kraft')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=27267728 AND mptpi.`type`=4417 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('mal_croppe','flou','Carton_brun','papier','environnement','plastique','cartonnette','Carton_gris','Teint_Dans_La_Masse','autre_refus','metal','kraft')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://marlene.fotonower.com/velours/27270837,27270838,27270839,27270840,27270841,27270842,27270843,27270844,27270845,27270846,27270847,27270848?tags=papier,Carton_brun,autre_refus,mal_croppe,plastique,environnement,Teint_Dans_La_Masse,kraft,flou,Carton_gris,metal,cartonnette Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1385110088, 1385110084, 1385110083, 1385110080, 1385106334, 1385106332, 1385106331, 1385106321] Looping around the photos to save general results len do output : 1 /27267728. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110088', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110084', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110083', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110080', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106334', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106332', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106331', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106321', None, None, None, None, None, '3784106') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 9 time used for this insertion : 0.016109704971313477 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.5829863548278809 time spend to save output : 0.016411542892456055 total time spend for step 10 : 1.599397897720337 step11:final Fri Sep 26 11:26:11 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1385110088: ('0.13238149096702773',), 1385110084: ('0.13238149096702773',), 1385110083: ('0.13238149096702773',), 1385110080: ('0.13238149096702773',), 1385106334: ('0.13238149096702773',), 1385106332: ('0.13238149096702773',), 1385106331: ('0.13238149096702773',), 1385106321: ('0.13238149096702773',)} new output for save of step final : {1385110088: ('0.13238149096702773',), 1385110084: ('0.13238149096702773',), 1385110083: ('0.13238149096702773',), 1385110080: ('0.13238149096702773',), 1385106334: ('0.13238149096702773',), 1385106332: ('0.13238149096702773',), 1385106331: ('0.13238149096702773',), 1385106321: ('0.13238149096702773',)} [1385110088, 1385110084, 1385110083, 1385110080, 1385106334, 1385106332, 1385106331, 1385106321] Looping around the photos to save general results len do output : 8 /1385110088.Didn't retrieve data . /1385110084.Didn't retrieve data . /1385110083.Didn't retrieve data . /1385110080.Didn't retrieve data . /1385106334.Didn't retrieve data . /1385106332.Didn't retrieve data . /1385106331.Didn't retrieve data . /1385106321.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110088', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110084', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110083', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110080', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106334', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106332', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106331', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106321', None, None, None, None, None, '3784106') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 24 time used for this insertion : 0.012585639953613281 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.13151121139526367 time spend to save output : 0.013039827346801758 total time spend for step 11 : 0.14455103874206543 step12:velours_tree Fri Sep 26 11:26:12 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.10073041915893555 time spend to save output : 3.814697265625e-05 total time spend for step 12 : 0.1007685661315918 step13:send_mail_cod Fri Sep 26 11:26:12 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 complete output_args for input 1 complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin in order to get the selector url, please entre the license of selector results_Auto_P27267728_26-09-2025_11_26_12.pdf 27270826 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette272708261758878772 27270827 imagette272708271758878773 27270828 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette272708281758878773 27270831 change filename to text .change filename to text .change filename to text .imagette272708311758878778 27270832 imagette272708321758878778 27270834 change filename to text .change filename to text .imagette272708341758878778 27270835 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette272708351758878779 27270825 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette272708251758878781 27270830 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette272708301758878784 27270833 change filename to text .change filename to text .change filename to text .imagette272708331758878785 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=27267728 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://marlene.fotonower.com/velours/27270837,27270838,27270839,27270840,27270841,27270842,27270843,27270844,27270845,27270846,27270847,27270848?tags=papier,Carton_brun,autre_refus,mal_croppe,plastique,environnement,Teint_Dans_La_Masse,kraft,flou,Carton_gris,metal,cartonnette your option no_mail is active, we will not send the real mail to your client args[1385110088] : ((1385110088, 0.06508842106999577, 2107752395), (1385110088, 0.11301307754241217, 492688767), '0.13238149096702773') We are sending mail with results at report@fotonower.com args[1385110084] : ((1385110084, 0.08805987152564133, 2107752395), (1385110084, -2.56660127615888, 492609224), '0.13238149096702773') We are sending mail with results at report@fotonower.com args[1385110083] : ((1385110083, 0.2735783793193647, 2107752395), (1385110083, -0.2650610727941277, 492688767), '0.13238149096702773') We are sending mail with results at report@fotonower.com args[1385110080] : ((1385110080, 0.03217041204923833, 2107752395), (1385110080, -0.028938604571266548, 492688767), '0.13238149096702773') We are sending mail with results at report@fotonower.com args[1385106334] : ((1385106334, 0.05046923357217197, 2107752395), (1385106334, -2.917658951981848, 492609224), '0.13238149096702773') We are sending mail with results at report@fotonower.com args[1385106332] : ((1385106332, 0.19842315240330086, 2107752395), (1385106332, -0.3036017932345212, 492688767), '0.13238149096702773') We are sending mail with results at report@fotonower.com args[1385106331] : ((1385106331, 0.11822340794571444, 2107752395), (1385106331, -0.23111732321729286, 492688767), '0.13238149096702773') We are sending mail with results at report@fotonower.com args[1385106321] : ((1385106321, 0.35547765371748624, 2107752395), (1385106321, -0.6932439128364671, 492688767), '0.13238149096702773') We are sending mail with results at report@fotonower.com refus_total : 0.13238149096702773 2022-04-13 10:29:59 0 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P27267728_26-09-2025_11_26_12.pdf results_Auto_P27267728_26-09-2025_11_26_12.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P27267728_26-09-2025_11_26_12.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('4433','27267728','results_Auto_P27267728_26-09-2025_11_26_12.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P27267728_26-09-2025_11_26_12.pdf','pdf','','1.54','0.13238149096702773') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1385110088, 1385110084, 1385110083, 1385110080, 1385106334, 1385106332, 1385106331, 1385106321] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110088', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110084', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110083', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110080', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106334', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106332', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106331', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106321', None, None, None, None, None, '3784106') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 8 time used for this insertion : 0.013095855712890625 save_final save missing photos in datou_result : time spend for datou_step_exec : 17.77632784843445 time spend to save output : 0.013722658157348633 total time spend for step 13 : 17.790050506591797 step14:split_time_score Fri Sep 26 11:26:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('14', 68),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 19092025 27267728 Nombre de photos uploadées : 68 / 23040 (0%) 19092025 27267728 Nombre de photos taguées (types de déchets): 0 / 68 (0%) 19092025 27267728 Nombre de photos taguées (volume) : 0 / 68 (0%) elapsed_time : load_data_split_time_score 1.9073486328125e-06 elapsed_time : order_list_meta_photo_and_scores 6.198883056640625e-06 ???????????????????????????????????????????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0031442642211914062 Catched exception ! Connect or reconnect ! Catched exception ! Connect or reconnect ! elapsed_time : insert_dashboard_record_day_entry 0.20870614051818848 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.16001606594613368 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P27066907_20-09-2025_10_10_49.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 27066907 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=27066907 AND mptpi.`type`=3726 To do Qualite : 0.16152385581089376 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P27267728_26-09-2025_11_26_12.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 27267728 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=27267728 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'19092025': {'nb_upload': 68, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1385110088, 1385110084, 1385110083, 1385110080, 1385106334, 1385106332, 1385106331, 1385106321] Looping around the photos to save general results len do output : 1 /27267728Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110088', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110084', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110083', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385110080', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106334', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106332', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106331', None, None, None, None, None, '3784106') ('4433', None, None, None, None, None, None, None, '3784106') ('4433', None, '1385106321', None, None, None, None, None, '3784106') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 9 time used for this insertion : 0.013261556625366211 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.5354692935943604 time spend to save output : 0.013439178466796875 total time spend for step 14 : 0.5489084720611572 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 2 len(input) + len(total_photo_id_missing) : 8 set_done_treatment 177.40user 147.26system 6:31.76elapsed 82%CPU (0avgtext+0avgdata 6247312maxresident)k 5749760inputs+288112outputs (91859major+20226333minor)pagefaults 0swaps