python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1192779 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 14>10 please execute split_portfolio.py -i 20351776 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2567679'] with mtr_portfolio_ids : ['20351776'] and first list_photo_ids : [] new path : /proc/1192779/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.9967384338378906 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 7 22:16:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-07 22:16:33.672874: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-07 22:16:33.699136: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-07 22:16:33.701137: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f4a94000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-07 22:16:33.701196: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-07 22:16:33.705174: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-07 22:16:33.839989: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x8d48ce0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-07 22:16:33.840043: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-07 22:16:33.841580: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 22:16:33.842008: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 22:16:33.845107: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 22:16:33.848114: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 22:16:33.848517: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 22:16:33.851207: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 22:16:33.852494: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 22:16:33.857477: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 22:16:33.858956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 22:16:33.859036: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 22:16:33.859794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-07 22:16:33.859810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-07 22:16:33.859820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-07 22:16:33.861112: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-07 22:16:34.158319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 22:16:34.158454: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 22:16:34.158478: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 22:16:34.158499: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 22:16:34.158518: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 22:16:34.158537: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 22:16:34.158556: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 22:16:34.158575: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 22:16:34.160290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 22:16:34.162044: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 22:16:34.162169: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 22:16:34.162192: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 22:16:34.162212: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 22:16:34.162231: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 22:16:34.162249: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 22:16:34.162268: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 22:16:34.162287: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 22:16:34.163987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 22:16:34.164031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-07 22:16:34.164042: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-07 22:16:34.164053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-07 22:16:34.165791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-07 22:16:44.971450: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 22:16:45.190478: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 27 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 34 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 26 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 26 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 27 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 32 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 Detection mask done ! Trying to reset tf kernel 1193250 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5485 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10774 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0005826950073242188 nb_pixel_total : 7152 time to create 1 rle with old method : 0.009865045547485352 length of segment : 140 time for calcul the mask position with numpy : 0.006300687789916992 nb_pixel_total : 135472 time to create 1 rle with old method : 0.16387462615966797 length of segment : 548 time for calcul the mask position with numpy : 0.000286102294921875 nb_pixel_total : 6647 time to create 1 rle with old method : 0.008440256118774414 length of segment : 110 time for calcul the mask position with numpy : 0.002202749252319336 nb_pixel_total : 72252 time to create 1 rle with old method : 0.07624387741088867 length of segment : 422 time for calcul the mask position with numpy : 0.00012922286987304688 nb_pixel_total : 2812 time to create 1 rle with old method : 0.0037012100219726562 length of segment : 54 time for calcul the mask position with numpy : 0.0008783340454101562 nb_pixel_total : 34673 time to create 1 rle with old method : 0.04401421546936035 length of segment : 164 time for calcul the mask position with numpy : 0.00024700164794921875 nb_pixel_total : 6816 time to create 1 rle with old method : 0.00901484489440918 length of segment : 113 time for calcul the mask position with numpy : 0.00033855438232421875 nb_pixel_total : 7086 time to create 1 rle with old method : 0.009240865707397461 length of segment : 162 time for calcul the mask position with numpy : 0.00040221214294433594 nb_pixel_total : 13059 time to create 1 rle with old method : 0.017107486724853516 length of segment : 90 time for calcul the mask position with numpy : 0.00036072731018066406 nb_pixel_total : 6560 time to create 1 rle with old method : 0.009417057037353516 length of segment : 114 time for calcul the mask position with numpy : 0.00037407875061035156 nb_pixel_total : 14778 time to create 1 rle with old method : 0.017707347869873047 length of segment : 194 time for calcul the mask position with numpy : 0.0010838508605957031 nb_pixel_total : 11761 time to create 1 rle with old method : 0.013386249542236328 length of segment : 147 time for calcul the mask position with numpy : 0.00010514259338378906 nb_pixel_total : 2174 time to create 1 rle with old method : 0.002507925033569336 length of segment : 76 time for calcul the mask position with numpy : 0.002109527587890625 nb_pixel_total : 95711 time to create 1 rle with old method : 0.10792851448059082 length of segment : 585 time for calcul the mask position with numpy : 0.0011925697326660156 nb_pixel_total : 40047 time to create 1 rle with old method : 0.05216407775878906 length of segment : 370 time for calcul the mask position with numpy : 0.0045545101165771484 nb_pixel_total : 175729 time to create 1 rle with new method : 0.01850295066833496 length of segment : 515 time for calcul the mask position with numpy : 0.00012063980102539062 nb_pixel_total : 2585 time to create 1 rle with old method : 0.0031080245971679688 length of segment : 63 time for calcul the mask position with numpy : 0.0008497238159179688 nb_pixel_total : 26594 time to create 1 rle with old method : 0.03052663803100586 length of segment : 273 time for calcul the mask position with numpy : 0.002151966094970703 nb_pixel_total : 103384 time to create 1 rle with old method : 0.11776566505432129 length of segment : 322 time for calcul the mask position with numpy : 0.0004398822784423828 nb_pixel_total : 18963 time to create 1 rle with old method : 0.0216829776763916 length of segment : 150 time for calcul the mask position with numpy : 0.0025420188903808594 nb_pixel_total : 106420 time to create 1 rle with old method : 0.12540650367736816 length of segment : 446 time for calcul the mask position with numpy : 0.00027871131896972656 nb_pixel_total : 5805 time to create 1 rle with old method : 0.008492231369018555 length of segment : 94 time for calcul the mask position with numpy : 0.0036878585815429688 nb_pixel_total : 114081 time to create 1 rle with old method : 0.14728021621704102 length of segment : 479 time for calcul the mask position with numpy : 0.0012390613555908203 nb_pixel_total : 51210 time to create 1 rle with old method : 0.06023001670837402 length of segment : 470 time for calcul the mask position with numpy : 0.0024976730346679688 nb_pixel_total : 90680 time to create 1 rle with old method : 0.10688090324401855 length of segment : 446 time for calcul the mask position with numpy : 0.0022644996643066406 nb_pixel_total : 94846 time to create 1 rle with old method : 0.10373544692993164 length of segment : 352 time for calcul the mask position with numpy : 0.0003914833068847656 nb_pixel_total : 15517 time to create 1 rle with old method : 0.020041942596435547 length of segment : 121 time for calcul the mask position with numpy : 0.00010037422180175781 nb_pixel_total : 1758 time to create 1 rle with old method : 0.002576589584350586 length of segment : 52 time for calcul the mask position with numpy : 0.00035762786865234375 nb_pixel_total : 8459 time to create 1 rle with old method : 0.010711669921875 length of segment : 142 time for calcul the mask position with numpy : 0.00038051605224609375 nb_pixel_total : 8553 time to create 1 rle with old method : 0.010308027267456055 length of segment : 116 time for calcul the mask position with numpy : 0.0002663135528564453 nb_pixel_total : 9048 time to create 1 rle with old method : 0.009890556335449219 length of segment : 105 time for calcul the mask position with numpy : 0.00035691261291503906 nb_pixel_total : 14035 time to create 1 rle with old method : 0.015421152114868164 length of segment : 221 time for calcul the mask position with numpy : 0.0010840892791748047 nb_pixel_total : 38807 time to create 1 rle with old method : 0.040926218032836914 length of segment : 330 time for calcul the mask position with numpy : 0.0016527175903320312 nb_pixel_total : 65854 time to create 1 rle with old method : 0.0733344554901123 length of segment : 421 time for calcul the mask position with numpy : 0.0005152225494384766 nb_pixel_total : 18842 time to create 1 rle with old method : 0.020816802978515625 length of segment : 160 time for calcul the mask position with numpy : 0.00015044212341308594 nb_pixel_total : 3364 time to create 1 rle with old method : 0.00374603271484375 length of segment : 89 time for calcul the mask position with numpy : 0.0004832744598388672 nb_pixel_total : 13246 time to create 1 rle with old method : 0.014406681060791016 length of segment : 185 time for calcul the mask position with numpy : 0.00022411346435546875 nb_pixel_total : 6246 time to create 1 rle with old method : 0.0068776607513427734 length of segment : 112 time for calcul the mask position with numpy : 0.00010824203491210938 nb_pixel_total : 2955 time to create 1 rle with old method : 0.0032753944396972656 length of segment : 84 time for calcul the mask position with numpy : 0.0013196468353271484 nb_pixel_total : 71156 time to create 1 rle with old method : 0.07589483261108398 length of segment : 349 time for calcul the mask position with numpy : 0.0009648799896240234 nb_pixel_total : 39186 time to create 1 rle with old method : 0.04213452339172363 length of segment : 416 time for calcul the mask position with numpy : 7.414817810058594e-05 nb_pixel_total : 1539 time to create 1 rle with old method : 0.0017991065979003906 length of segment : 40 time for calcul the mask position with numpy : 0.0039484500885009766 nb_pixel_total : 163708 time to create 1 rle with new method : 0.01499319076538086 length of segment : 509 time for calcul the mask position with numpy : 0.0013000965118408203 nb_pixel_total : 44870 time to create 1 rle with old method : 0.04908323287963867 length of segment : 460 time for calcul the mask position with numpy : 0.0004968643188476562 nb_pixel_total : 20260 time to create 1 rle with old method : 0.021660327911376953 length of segment : 157 time for calcul the mask position with numpy : 0.0004622936248779297 nb_pixel_total : 17388 time to create 1 rle with old method : 0.0188143253326416 length of segment : 209 time for calcul the mask position with numpy : 9.894371032714844e-05 nb_pixel_total : 2119 time to create 1 rle with old method : 0.002402782440185547 length of segment : 42 time for calcul the mask position with numpy : 0.000499725341796875 nb_pixel_total : 18663 time to create 1 rle with old method : 0.02013707160949707 length of segment : 135 time for calcul the mask position with numpy : 0.0006041526794433594 nb_pixel_total : 14516 time to create 1 rle with old method : 0.019261598587036133 length of segment : 130 time for calcul the mask position with numpy : 0.017005205154418945 nb_pixel_total : 354041 time to create 1 rle with new method : 0.03218650817871094 length of segment : 506 time for calcul the mask position with numpy : 0.0013163089752197266 nb_pixel_total : 28162 time to create 1 rle with old method : 0.03387284278869629 length of segment : 300 time for calcul the mask position with numpy : 8.225440979003906e-05 nb_pixel_total : 1553 time to create 1 rle with old method : 0.001973390579223633 length of segment : 43 time for calcul the mask position with numpy : 0.00579524040222168 nb_pixel_total : 300707 time to create 1 rle with new method : 0.034474849700927734 length of segment : 332 time for calcul the mask position with numpy : 0.0004191398620605469 nb_pixel_total : 16189 time to create 1 rle with old method : 0.01971268653869629 length of segment : 181 time for calcul the mask position with numpy : 0.0002522468566894531 nb_pixel_total : 12270 time to create 1 rle with old method : 0.01392507553100586 length of segment : 137 time for calcul the mask position with numpy : 0.0002551078796386719 nb_pixel_total : 13355 time to create 1 rle with old method : 0.01558542251586914 length of segment : 148 time for calcul the mask position with numpy : 0.00010204315185546875 nb_pixel_total : 2414 time to create 1 rle with old method : 0.003164529800415039 length of segment : 61 time for calcul the mask position with numpy : 0.000701904296875 nb_pixel_total : 21362 time to create 1 rle with old method : 0.023877382278442383 length of segment : 130 time for calcul the mask position with numpy : 0.0004584789276123047 nb_pixel_total : 23734 time to create 1 rle with old method : 0.027660131454467773 length of segment : 308 time for calcul the mask position with numpy : 0.0008955001831054688 nb_pixel_total : 22965 time to create 1 rle with old method : 0.025572538375854492 length of segment : 210 time for calcul the mask position with numpy : 0.0002560615539550781 nb_pixel_total : 3998 time to create 1 rle with old method : 0.004479646682739258 length of segment : 100 time for calcul the mask position with numpy : 0.00043129920959472656 nb_pixel_total : 10862 time to create 1 rle with old method : 0.012566328048706055 length of segment : 108 time for calcul the mask position with numpy : 0.0015213489532470703 nb_pixel_total : 26460 time to create 1 rle with old method : 0.033156394958496094 length of segment : 312 time for calcul the mask position with numpy : 0.0032875537872314453 nb_pixel_total : 72263 time to create 1 rle with old method : 0.07522821426391602 length of segment : 434 time for calcul the mask position with numpy : 0.00023031234741210938 nb_pixel_total : 3436 time to create 1 rle with old method : 0.004073381423950195 length of segment : 56 time for calcul the mask position with numpy : 0.0002887248992919922 nb_pixel_total : 4937 time to create 1 rle with old method : 0.005520820617675781 length of segment : 112 time for calcul the mask position with numpy : 0.0004291534423828125 nb_pixel_total : 11185 time to create 1 rle with old method : 0.012859582901000977 length of segment : 164 time for calcul the mask position with numpy : 0.0016374588012695312 nb_pixel_total : 30139 time to create 1 rle with old method : 0.03461480140686035 length of segment : 346 time for calcul the mask position with numpy : 0.0015194416046142578 nb_pixel_total : 45736 time to create 1 rle with old method : 0.05254983901977539 length of segment : 327 time for calcul the mask position with numpy : 0.009843826293945312 nb_pixel_total : 392273 time to create 1 rle with new method : 0.02897191047668457 length of segment : 816 time for calcul the mask position with numpy : 0.0009675025939941406 nb_pixel_total : 22449 time to create 1 rle with old method : 0.02325725555419922 length of segment : 254 time for calcul the mask position with numpy : 0.0002560615539550781 nb_pixel_total : 15839 time to create 1 rle with old method : 0.017090797424316406 length of segment : 166 time for calcul the mask position with numpy : 0.0005457401275634766 nb_pixel_total : 15340 time to create 1 rle with old method : 0.016492843627929688 length of segment : 158 time for calcul the mask position with numpy : 0.0008635520935058594 nb_pixel_total : 15509 time to create 1 rle with old method : 0.016872882843017578 length of segment : 202 time for calcul the mask position with numpy : 0.0011370182037353516 nb_pixel_total : 31804 time to create 1 rle with old method : 0.035974979400634766 length of segment : 294 time for calcul the mask position with numpy : 0.0016007423400878906 nb_pixel_total : 55849 time to create 1 rle with old method : 0.06399130821228027 length of segment : 209 time for calcul the mask position with numpy : 0.0006549358367919922 nb_pixel_total : 21909 time to create 1 rle with old method : 0.024729490280151367 length of segment : 107 time for calcul the mask position with numpy : 0.00014519691467285156 nb_pixel_total : 1118 time to create 1 rle with old method : 0.0013284683227539062 length of segment : 59 time for calcul the mask position with numpy : 0.001013040542602539 nb_pixel_total : 25970 time to create 1 rle with old method : 0.02744770050048828 length of segment : 187 time for calcul the mask position with numpy : 0.0007719993591308594 nb_pixel_total : 15636 time to create 1 rle with old method : 0.01624774932861328 length of segment : 313 time for calcul the mask position with numpy : 0.0013685226440429688 nb_pixel_total : 43755 time to create 1 rle with old method : 0.04807424545288086 length of segment : 274 time for calcul the mask position with numpy : 0.0003261566162109375 nb_pixel_total : 2724 time to create 1 rle with old method : 0.0034210681915283203 length of segment : 90 time for calcul the mask position with numpy : 0.0005846023559570312 nb_pixel_total : 11440 time to create 1 rle with old method : 0.012871742248535156 length of segment : 179 time for calcul the mask position with numpy : 0.00020003318786621094 nb_pixel_total : 1729 time to create 1 rle with old method : 0.0022149085998535156 length of segment : 80 time for calcul the mask position with numpy : 0.0002732276916503906 nb_pixel_total : 5965 time to create 1 rle with old method : 0.00718998908996582 length of segment : 69 time for calcul the mask position with numpy : 0.00023436546325683594 nb_pixel_total : 3610 time to create 1 rle with old method : 0.004203319549560547 length of segment : 86 time for calcul the mask position with numpy : 0.002984762191772461 nb_pixel_total : 83013 time to create 1 rle with old method : 0.11413741111755371 length of segment : 491 time for calcul the mask position with numpy : 0.00027489662170410156 nb_pixel_total : 2977 time to create 1 rle with old method : 0.0035822391510009766 length of segment : 113 time for calcul the mask position with numpy : 0.004018068313598633 nb_pixel_total : 105955 time to create 1 rle with old method : 0.11493945121765137 length of segment : 695 time for calcul the mask position with numpy : 0.0013496875762939453 nb_pixel_total : 67182 time to create 1 rle with old method : 0.07310700416564941 length of segment : 419 time for calcul the mask position with numpy : 0.001125335693359375 nb_pixel_total : 23470 time to create 1 rle with old method : 0.02649521827697754 length of segment : 237 time for calcul the mask position with numpy : 0.0002453327178955078 nb_pixel_total : 4105 time to create 1 rle with old method : 0.0048427581787109375 length of segment : 83 time for calcul the mask position with numpy : 0.0005600452423095703 nb_pixel_total : 18148 time to create 1 rle with old method : 0.02115607261657715 length of segment : 206 time for calcul the mask position with numpy : 0.0003311634063720703 nb_pixel_total : 11931 time to create 1 rle with old method : 0.013592958450317383 length of segment : 204 time for calcul the mask position with numpy : 0.002557039260864258 nb_pixel_total : 86314 time to create 1 rle with old method : 0.09706616401672363 length of segment : 455 time for calcul the mask position with numpy : 0.0006973743438720703 nb_pixel_total : 14011 time to create 1 rle with old method : 0.01621270179748535 length of segment : 198 time for calcul the mask position with numpy : 0.00022673606872558594 nb_pixel_total : 3726 time to create 1 rle with old method : 0.004305839538574219 length of segment : 68 time for calcul the mask position with numpy : 0.0026726722717285156 nb_pixel_total : 27203 time to create 1 rle with old method : 0.030907154083251953 length of segment : 269 time for calcul the mask position with numpy : 0.00033926963806152344 nb_pixel_total : 3849 time to create 1 rle with old method : 0.004563570022583008 length of segment : 153 time for calcul the mask position with numpy : 0.0011756420135498047 nb_pixel_total : 29950 time to create 1 rle with old method : 0.03424549102783203 length of segment : 309 time for calcul the mask position with numpy : 0.0005402565002441406 nb_pixel_total : 9754 time to create 1 rle with old method : 0.011578798294067383 length of segment : 164 time for calcul the mask position with numpy : 0.000518798828125 nb_pixel_total : 14155 time to create 1 rle with old method : 0.017111539840698242 length of segment : 113 time for calcul the mask position with numpy : 0.0027844905853271484 nb_pixel_total : 59283 time to create 1 rle with old method : 0.06731677055358887 length of segment : 288 time for calcul the mask position with numpy : 0.0017273426055908203 nb_pixel_total : 37308 time to create 1 rle with old method : 0.04266357421875 length of segment : 411 time for calcul the mask position with numpy : 0.0007264614105224609 nb_pixel_total : 17260 time to create 1 rle with old method : 0.021004915237426758 length of segment : 94 time for calcul the mask position with numpy : 0.0016732215881347656 nb_pixel_total : 38431 time to create 1 rle with old method : 0.04408144950866699 length of segment : 318 time for calcul the mask position with numpy : 0.0023479461669921875 nb_pixel_total : 69240 time to create 1 rle with old method : 0.0780482292175293 length of segment : 340 time for calcul the mask position with numpy : 0.0036509037017822266 nb_pixel_total : 82452 time to create 1 rle with old method : 0.09194779396057129 length of segment : 548 time for calcul the mask position with numpy : 0.002331256866455078 nb_pixel_total : 25272 time to create 1 rle with old method : 0.028927087783813477 length of segment : 399 time for calcul the mask position with numpy : 0.0032896995544433594 nb_pixel_total : 63715 time to create 1 rle with old method : 0.07019424438476562 length of segment : 288 time for calcul the mask position with numpy : 0.0016913414001464844 nb_pixel_total : 56358 time to create 1 rle with old method : 0.06246685981750488 length of segment : 313 time for calcul the mask position with numpy : 0.0115203857421875 nb_pixel_total : 269949 time to create 1 rle with new method : 0.14582586288452148 length of segment : 970 time for calcul the mask position with numpy : 0.004041194915771484 nb_pixel_total : 119898 time to create 1 rle with old method : 0.13682866096496582 length of segment : 440 time for calcul the mask position with numpy : 0.00023865699768066406 nb_pixel_total : 3853 time to create 1 rle with old method : 0.004875898361206055 length of segment : 40 time for calcul the mask position with numpy : 0.002307891845703125 nb_pixel_total : 61943 time to create 1 rle with old method : 0.0702822208404541 length of segment : 286 time for calcul the mask position with numpy : 0.004757881164550781 nb_pixel_total : 224497 time to create 1 rle with new method : 0.026209592819213867 length of segment : 821 time for calcul the mask position with numpy : 0.00021982192993164062 nb_pixel_total : 2260 time to create 1 rle with old method : 0.002965688705444336 length of segment : 64 time for calcul the mask position with numpy : 0.003453969955444336 nb_pixel_total : 126222 time to create 1 rle with old method : 0.13765835762023926 length of segment : 488 time for calcul the mask position with numpy : 0.0033042430877685547 nb_pixel_total : 68903 time to create 1 rle with old method : 0.07510900497436523 length of segment : 533 time for calcul the mask position with numpy : 0.005483865737915039 nb_pixel_total : 95941 time to create 1 rle with old method : 0.1107168197631836 length of segment : 742 time for calcul the mask position with numpy : 0.0013382434844970703 nb_pixel_total : 36691 time to create 1 rle with old method : 0.043747663497924805 length of segment : 304 time for calcul the mask position with numpy : 0.0012731552124023438 nb_pixel_total : 25748 time to create 1 rle with old method : 0.028849124908447266 length of segment : 459 time for calcul the mask position with numpy : 0.003015756607055664 nb_pixel_total : 115526 time to create 1 rle with old method : 0.12441277503967285 length of segment : 519 time for calcul the mask position with numpy : 0.0006167888641357422 nb_pixel_total : 16026 time to create 1 rle with old method : 0.018716096878051758 length of segment : 238 time for calcul the mask position with numpy : 0.00501251220703125 nb_pixel_total : 157422 time to create 1 rle with new method : 0.011817455291748047 length of segment : 628 time for calcul the mask position with numpy : 0.005035877227783203 nb_pixel_total : 179184 time to create 1 rle with new method : 0.016247034072875977 length of segment : 861 time for calcul the mask position with numpy : 0.0010066032409667969 nb_pixel_total : 46915 time to create 1 rle with old method : 0.054904937744140625 length of segment : 543 time for calcul the mask position with numpy : 0.0013616085052490234 nb_pixel_total : 36012 time to create 1 rle with old method : 0.040457725524902344 length of segment : 309 time for calcul the mask position with numpy : 0.0046234130859375 nb_pixel_total : 62870 time to create 1 rle with old method : 0.07472562789916992 length of segment : 459 time for calcul the mask position with numpy : 0.0033571720123291016 nb_pixel_total : 91240 time to create 1 rle with old method : 0.10532927513122559 length of segment : 525 time for calcul the mask position with numpy : 0.00028014183044433594 nb_pixel_total : 7035 time to create 1 rle with old method : 0.008696794509887695 length of segment : 81 time for calcul the mask position with numpy : 0.02062821388244629 nb_pixel_total : 535179 time to create 1 rle with new method : 0.09400415420532227 length of segment : 1366 time for calcul the mask position with numpy : 0.0016696453094482422 nb_pixel_total : 58505 time to create 1 rle with old method : 0.06720590591430664 length of segment : 376 time for calcul the mask position with numpy : 0.00013566017150878906 nb_pixel_total : 3135 time to create 1 rle with old method : 0.004517078399658203 length of segment : 58 time for calcul the mask position with numpy : 0.0004286766052246094 nb_pixel_total : 13813 time to create 1 rle with old method : 0.016226768493652344 length of segment : 135 time for calcul the mask position with numpy : 0.00098419189453125 nb_pixel_total : 46580 time to create 1 rle with old method : 0.05344438552856445 length of segment : 264 time for calcul the mask position with numpy : 0.0010309219360351562 nb_pixel_total : 48073 time to create 1 rle with old method : 0.053549766540527344 length of segment : 247 time for calcul the mask position with numpy : 0.002276897430419922 nb_pixel_total : 147430 time to create 1 rle with old method : 0.16182732582092285 length of segment : 443 time for calcul the mask position with numpy : 0.0020356178283691406 nb_pixel_total : 89776 time to create 1 rle with old method : 0.10020256042480469 length of segment : 507 time for calcul the mask position with numpy : 0.0012671947479248047 nb_pixel_total : 83957 time to create 1 rle with old method : 0.0906674861907959 length of segment : 512 time for calcul the mask position with numpy : 0.0010104179382324219 nb_pixel_total : 48391 time to create 1 rle with old method : 0.056104183197021484 length of segment : 346 time for calcul the mask position with numpy : 0.0009543895721435547 nb_pixel_total : 41649 time to create 1 rle with old method : 0.04768037796020508 length of segment : 215 time for calcul the mask position with numpy : 9.489059448242188e-05 nb_pixel_total : 2096 time to create 1 rle with old method : 0.0027358531951904297 length of segment : 52 time for calcul the mask position with numpy : 0.0026865005493164062 nb_pixel_total : 171718 time to create 1 rle with new method : 0.006186246871948242 length of segment : 265 time for calcul the mask position with numpy : 0.0023717880249023438 nb_pixel_total : 63873 time to create 1 rle with old method : 0.07110881805419922 length of segment : 531 time for calcul the mask position with numpy : 0.0005977153778076172 nb_pixel_total : 36562 time to create 1 rle with old method : 0.04154610633850098 length of segment : 170 time for calcul the mask position with numpy : 0.0003802776336669922 nb_pixel_total : 22346 time to create 1 rle with old method : 0.026849031448364258 length of segment : 137 time for calcul the mask position with numpy : 0.0013616085052490234 nb_pixel_total : 52683 time to create 1 rle with old method : 0.05869460105895996 length of segment : 351 time for calcul the mask position with numpy : 0.0021948814392089844 nb_pixel_total : 87624 time to create 1 rle with old method : 0.09860014915466309 length of segment : 453 time for calcul the mask position with numpy : 0.0002543926239013672 nb_pixel_total : 3635 time to create 1 rle with old method : 0.004149436950683594 length of segment : 78 time for calcul the mask position with numpy : 0.0014331340789794922 nb_pixel_total : 43601 time to create 1 rle with old method : 0.04828166961669922 length of segment : 200 time for calcul the mask position with numpy : 0.00017642974853515625 nb_pixel_total : 6150 time to create 1 rle with old method : 0.00731658935546875 length of segment : 106 time for calcul the mask position with numpy : 0.0013628005981445312 nb_pixel_total : 65865 time to create 1 rle with old method : 0.07145261764526367 length of segment : 430 time for calcul the mask position with numpy : 0.0002315044403076172 nb_pixel_total : 7751 time to create 1 rle with old method : 0.008951663970947266 length of segment : 119 time for calcul the mask position with numpy : 0.0037398338317871094 nb_pixel_total : 139320 time to create 1 rle with old method : 0.15895318984985352 length of segment : 396 time for calcul the mask position with numpy : 8.511543273925781e-05 nb_pixel_total : 1592 time to create 1 rle with old method : 0.0018503665924072266 length of segment : 59 time for calcul the mask position with numpy : 0.0010416507720947266 nb_pixel_total : 25420 time to create 1 rle with old method : 0.02922511100769043 length of segment : 362 time for calcul the mask position with numpy : 0.00151824951171875 nb_pixel_total : 73955 time to create 1 rle with old method : 0.08050394058227539 length of segment : 524 time for calcul the mask position with numpy : 0.00439143180847168 nb_pixel_total : 190301 time to create 1 rle with new method : 0.010995864868164062 length of segment : 525 time for calcul the mask position with numpy : 8.463859558105469e-05 nb_pixel_total : 2139 time to create 1 rle with old method : 0.0025916099548339844 length of segment : 41 time for calcul the mask position with numpy : 0.002974271774291992 nb_pixel_total : 150204 time to create 1 rle with new method : 0.006794452667236328 length of segment : 549 time for calcul the mask position with numpy : 0.0033659934997558594 nb_pixel_total : 196967 time to create 1 rle with new method : 0.010473251342773438 length of segment : 453 time for calcul the mask position with numpy : 0.0005140304565429688 nb_pixel_total : 19058 time to create 1 rle with old method : 0.021332263946533203 length of segment : 207 time for calcul the mask position with numpy : 0.002084493637084961 nb_pixel_total : 70317 time to create 1 rle with old method : 0.07808256149291992 length of segment : 286 time for calcul the mask position with numpy : 0.003155231475830078 nb_pixel_total : 190426 time to create 1 rle with new method : 0.0075609683990478516 length of segment : 524 time for calcul the mask position with numpy : 0.0002880096435546875 nb_pixel_total : 6453 time to create 1 rle with old method : 0.007508754730224609 length of segment : 100 time for calcul the mask position with numpy : 0.000728607177734375 nb_pixel_total : 42164 time to create 1 rle with old method : 0.04793071746826172 length of segment : 276 time for calcul the mask position with numpy : 0.00022530555725097656 nb_pixel_total : 7999 time to create 1 rle with old method : 0.009662151336669922 length of segment : 72 time for calcul the mask position with numpy : 0.0008819103240966797 nb_pixel_total : 33740 time to create 1 rle with old method : 0.039217233657836914 length of segment : 199 time for calcul the mask position with numpy : 0.0019643306732177734 nb_pixel_total : 82611 time to create 1 rle with old method : 0.0915684700012207 length of segment : 569 time for calcul the mask position with numpy : 0.00682377815246582 nb_pixel_total : 324372 time to create 1 rle with new method : 0.013671159744262695 length of segment : 417 time for calcul the mask position with numpy : 0.0007987022399902344 nb_pixel_total : 39246 time to create 1 rle with old method : 0.04439878463745117 length of segment : 209 time for calcul the mask position with numpy : 0.0014843940734863281 nb_pixel_total : 66508 time to create 1 rle with old method : 0.07377839088439941 length of segment : 408 time for calcul the mask position with numpy : 0.0015172958374023438 nb_pixel_total : 75773 time to create 1 rle with old method : 0.08598971366882324 length of segment : 495 time for calcul the mask position with numpy : 0.0006592273712158203 nb_pixel_total : 23410 time to create 1 rle with old method : 0.027014970779418945 length of segment : 187 time for calcul the mask position with numpy : 0.004515886306762695 nb_pixel_total : 328645 time to create 1 rle with new method : 0.016767024993896484 length of segment : 498 time for calcul the mask position with numpy : 0.0007555484771728516 nb_pixel_total : 42275 time to create 1 rle with old method : 0.04723644256591797 length of segment : 276 time for calcul the mask position with numpy : 0.0004432201385498047 nb_pixel_total : 12948 time to create 1 rle with old method : 0.015174150466918945 length of segment : 181 time for calcul the mask position with numpy : 0.0032982826232910156 nb_pixel_total : 112208 time to create 1 rle with old method : 0.1210947036743164 length of segment : 450 time for calcul the mask position with numpy : 0.002168416976928711 nb_pixel_total : 78331 time to create 1 rle with old method : 0.08559775352478027 length of segment : 510 time for calcul the mask position with numpy : 0.002280712127685547 nb_pixel_total : 87002 time to create 1 rle with old method : 0.09432792663574219 length of segment : 352 time for calcul the mask position with numpy : 0.0016438961029052734 nb_pixel_total : 44280 time to create 1 rle with old method : 0.06588554382324219 length of segment : 347 time for calcul the mask position with numpy : 0.009823799133300781 nb_pixel_total : 347097 time to create 1 rle with new method : 0.1691906452178955 length of segment : 854 time for calcul the mask position with numpy : 0.0002872943878173828 nb_pixel_total : 8143 time to create 1 rle with old method : 0.009566783905029297 length of segment : 116 time for calcul the mask position with numpy : 0.00013899803161621094 nb_pixel_total : 2052 time to create 1 rle with old method : 0.0025475025177001953 length of segment : 38 time for calcul the mask position with numpy : 0.0002636909484863281 nb_pixel_total : 6366 time to create 1 rle with old method : 0.0078277587890625 length of segment : 84 time for calcul the mask position with numpy : 0.005384683609008789 nb_pixel_total : 347548 time to create 1 rle with new method : 0.025938987731933594 length of segment : 938 time for calcul the mask position with numpy : 0.0024733543395996094 nb_pixel_total : 84624 time to create 1 rle with old method : 0.09327864646911621 length of segment : 431 time for calcul the mask position with numpy : 0.00013780593872070312 nb_pixel_total : 2539 time to create 1 rle with old method : 0.0030982494354248047 length of segment : 44 time for calcul the mask position with numpy : 0.0003571510314941406 nb_pixel_total : 19925 time to create 1 rle with old method : 0.021419525146484375 length of segment : 153 time for calcul the mask position with numpy : 0.00107574462890625 nb_pixel_total : 28612 time to create 1 rle with old method : 0.03159689903259277 length of segment : 257 time for calcul the mask position with numpy : 0.0005517005920410156 nb_pixel_total : 33280 time to create 1 rle with old method : 0.03798651695251465 length of segment : 300 time for calcul the mask position with numpy : 0.00016498565673828125 nb_pixel_total : 2809 time to create 1 rle with old method : 0.003189563751220703 length of segment : 87 time for calcul the mask position with numpy : 0.0006947517395019531 nb_pixel_total : 24883 time to create 1 rle with old method : 0.027315139770507812 length of segment : 222 time for calcul the mask position with numpy : 0.0006833076477050781 nb_pixel_total : 27898 time to create 1 rle with old method : 0.03084254264831543 length of segment : 193 time for calcul the mask position with numpy : 0.0005578994750976562 nb_pixel_total : 17196 time to create 1 rle with old method : 0.020107507705688477 length of segment : 119 time for calcul the mask position with numpy : 0.0005772113800048828 nb_pixel_total : 18606 time to create 1 rle with old method : 0.021220922470092773 length of segment : 201 time for calcul the mask position with numpy : 0.0002028942108154297 nb_pixel_total : 3317 time to create 1 rle with old method : 0.0038285255432128906 length of segment : 94 time for calcul the mask position with numpy : 0.002967357635498047 nb_pixel_total : 138875 time to create 1 rle with old method : 0.14781475067138672 length of segment : 606 time for calcul the mask position with numpy : 0.0002205371856689453 nb_pixel_total : 7301 time to create 1 rle with old method : 0.008306264877319336 length of segment : 74 time for calcul the mask position with numpy : 0.0004284381866455078 nb_pixel_total : 12054 time to create 1 rle with old method : 0.013904809951782227 length of segment : 151 time for calcul the mask position with numpy : 0.0013837814331054688 nb_pixel_total : 41401 time to create 1 rle with old method : 0.04650425910949707 length of segment : 226 time for calcul the mask position with numpy : 9.632110595703125e-05 nb_pixel_total : 2493 time to create 1 rle with old method : 0.0028853416442871094 length of segment : 92 time for calcul the mask position with numpy : 0.0003273487091064453 nb_pixel_total : 11088 time to create 1 rle with old method : 0.012488842010498047 length of segment : 154 time for calcul the mask position with numpy : 0.0009789466857910156 nb_pixel_total : 33360 time to create 1 rle with old method : 0.03863859176635742 length of segment : 146 time for calcul the mask position with numpy : 0.00039386749267578125 nb_pixel_total : 5629 time to create 1 rle with old method : 0.0070037841796875 length of segment : 125 time for calcul the mask position with numpy : 0.00029158592224121094 nb_pixel_total : 5789 time to create 1 rle with old method : 0.006796121597290039 length of segment : 81 time for calcul the mask position with numpy : 0.00036144256591796875 nb_pixel_total : 4831 time to create 1 rle with old method : 0.0058667659759521484 length of segment : 103 time spent for convertir_results : 17.272778272628784 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 208 chid ids of type : 3663 Number RLEs to save : 58550 save missing photos in datou_result : time spend for datou_step_exec : 78.35970616340637 time spend to save output : 4.184997797012329 total time spend for step 1 : 82.5447039604187 step2:crop_condition Fri Feb 7 22:17:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 208 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 13 About to insert : list_path_to_insert length 13 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 16 About to insert : list_path_to_insert length 16 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 65 About to insert : list_path_to_insert length 65 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335474142, 1335474138, 1335472989, 1335472873, 1335472785, 1335460491, 1335460487, 1335460482, 1335460478, 1335460424] Looping around the photos to save general results len do output : 116 /-3660544816Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544845Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544865Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544870Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544876Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544895Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544916Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544910Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544912Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544932Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544950Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544943Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544969Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544863Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544858Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544878Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544881Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544900Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544930Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544913Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544915Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544931Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544938Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544940Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544933Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544936Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544937Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544998Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544997Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544826Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544831Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544904Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544955Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544974Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544987Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544982Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660545001Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660545012Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544822Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544814Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544834Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544824Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544829Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544835Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544812Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544828Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544855Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544850Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544864Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544843Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544844Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544851Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544836Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544853Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544856Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544879Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544882Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544884Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544874Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544869Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544901Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544889Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544899Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544905Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544891Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544923Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544926Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544918Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544919Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544920Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544921Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544928Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544942Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544939Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544935Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544948Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544958Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544949Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544951Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544952Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544946Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544983Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544988Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544963Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544973Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544975Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544977Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544965Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544968Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544985Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544986Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544979Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544971Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544959Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544980Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544991Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544990Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544989Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544992Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660545009Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660545015Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660545007Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660545004Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544825Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544813Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544848Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544885Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544893Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544908Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544934Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544956Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544947Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544964Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544972Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544860Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660544886Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474142', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474138', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472989', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472873', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472785', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460491', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460487', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460482', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460478', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460424', None, None, None, None, None, '2567679') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 358 time used for this insertion : 0.06119418144226074 save_final save missing photos in datou_result : time spend for datou_step_exec : 47.07465600967407 time spend to save output : 0.06469440460205078 total time spend for step 2 : 47.13935041427612 step3:thcl Fri Feb 7 22:18:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.01595020294189453 time to convert the images to numpy array : 0.39122939109802246 time to import caffe and check if the image exist : 0.00928354263305664 time to convert the images to numpy array : 0.5890069007873535 time to import caffe and check if the image exist : 0.020390033721923828 time to convert the images to numpy array : 0.6073968410491943 time to import caffe and check if the image exist : 0.020217180252075195 time to convert the images to numpy array : 0.6214654445648193 time to import caffe and check if the image exist : 0.013656854629516602 time to convert the images to numpy array : 0.6389138698577881 time to import caffe and check if the image exist : 0.015676498413085938 time to convert the images to numpy array : 0.7557733058929443 time to import caffe and check if the image exist : 0.017279386520385742 time to convert the images to numpy array : 0.7580537796020508 time to import caffe and check if the image exist : 0.013786077499389648 time to convert the images to numpy array : 0.764319658279419 time to import caffe and check if the image exist : 0.019086599349975586 time to convert the images to numpy array : 0.7746658325195312 time to import caffe and check if the image exist : 0.0241851806640625 time to convert the images to numpy array : 0.7857611179351807 total time to convert the images to numpy array : 1.147155523300171 list photo_ids error: [] list photo_ids correct : [-3660544816, -3660544845, -3660544865, -3660544870, -3660544876, -3660544895, -3660544916, -3660544910, -3660544912, -3660544932, -3660544950, -3660544943, -3660544908, -3660544934, -3660544956, -3660544947, -3660544964, -3660544972, -3660544860, -3660544886, -3660544990, -3660544989, -3660544992, -3660545009, -3660545015, -3660545007, -3660545004, -3660544825, -3660544813, -3660544848, -3660544885, -3660544893, -3660544864, -3660544843, -3660544844, -3660544851, -3660544836, -3660544853, -3660544856, -3660544879, -3660544882, -3660544884, -3660544874, -3660544869, -3660545001, -3660545012, -3660544822, -3660544814, -3660544834, -3660544824, -3660544829, -3660544835, -3660544812, -3660544828, -3660544855, -3660544850, -3660544973, -3660544975, -3660544977, -3660544965, -3660544968, -3660544985, -3660544986, -3660544979, -3660544971, -3660544959, -3660544980, -3660544991, -3660544942, -3660544939, -3660544935, -3660544948, -3660544958, -3660544949, -3660544951, -3660544952, -3660544946, -3660544983, -3660544988, -3660544963, -3660544969, -3660544863, -3660544858, -3660544878, -3660544881, -3660544900, -3660544930, -3660544913, -3660544915, -3660544931, -3660544938, -3660544940, -3660544901, -3660544889, -3660544899, -3660544905, -3660544891, -3660544923, -3660544926, -3660544918, -3660544919, -3660544920, -3660544921, -3660544928, -3660544933, -3660544936, -3660544937, -3660544998, -3660544997, -3660544826, -3660544831, -3660544904, -3660544955, -3660544974, -3660544987, -3660544982] number of photos to traite : 116 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 9847 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5266 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.4364337921142578 time used to do the prediction : 0.32715916633605957 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5264 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5264 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.478400468826294 time used to do the prediction : 0.1757347583770752 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.621246337890625e-05 save missing photos in datou_result : time spend for datou_step_exec : 11.60072112083435 time spend to save output : 0.0011401176452636719 total time spend for step 3 : 11.601861238479614 step4:merge_mask_thcl_custom Fri Feb 7 22:18:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 208 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1335474142, 1335474138, 1335472989, 1335472873, 1335472785, 1335460491, 1335460487, 1335460482, 1335460478, 1335460424] Looping around the photos to save general results len do output : 10 /1335474142Didn't retrieve data .Didn't retrieve data . /1335474138Didn't retrieve data .Didn't retrieve data . /1335472989Didn't retrieve data .Didn't retrieve data . /1335472873Didn't retrieve data .Didn't retrieve data . /1335472785Didn't retrieve data .Didn't retrieve data . /1335460491Didn't retrieve data .Didn't retrieve data . /1335460487Didn't retrieve data .Didn't retrieve data . /1335460482Didn't retrieve data .Didn't retrieve data . /1335460478Didn't retrieve data .Didn't retrieve data . /1335460424Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474142', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474138', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472989', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472873', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472785', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460491', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460487', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460482', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460478', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460424', None, None, None, None, None, '2567679') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01668715476989746 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10648536682128906 time spend to save output : 0.01709294319152832 total time spend for step 4 : 0.12357831001281738 step5:rle_unique_nms_with_priority Fri Feb 7 22:18:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 23 nb_hashtags : 8 time to prepare the origin masks : 10.82041621208191 time for calcul the mask position with numpy : 0.3053297996520996 nb_pixel_total : 6035180 time to create 1 rle with new method : 0.7718620300292969 time for calcul the mask position with numpy : 0.03815507888793945 nb_pixel_total : 11595 time to create 1 rle with old method : 0.012259721755981445 time for calcul the mask position with numpy : 0.0382082462310791 nb_pixel_total : 6633 time to create 1 rle with old method : 0.0076296329498291016 time for calcul the mask position with numpy : 0.03786778450012207 nb_pixel_total : 106019 time to create 1 rle with old method : 0.10890674591064453 time for calcul the mask position with numpy : 0.025244712829589844 nb_pixel_total : 4901 time to create 1 rle with old method : 0.005614757537841797 time for calcul the mask position with numpy : 0.022760391235351562 nb_pixel_total : 39883 time to create 1 rle with old method : 0.04125094413757324 time for calcul the mask position with numpy : 0.02382946014404297 nb_pixel_total : 2562 time to create 1 rle with old method : 0.002788066864013672 time for calcul the mask position with numpy : 0.023254871368408203 nb_pixel_total : 134983 time to create 1 rle with old method : 0.13714170455932617 time for calcul the mask position with numpy : 0.0231473445892334 nb_pixel_total : 26513 time to create 1 rle with old method : 0.028077125549316406 time for calcul the mask position with numpy : 0.023499488830566406 nb_pixel_total : 103091 time to create 1 rle with old method : 0.10939693450927734 time for calcul the mask position with numpy : 0.022827863693237305 nb_pixel_total : 2163 time to create 1 rle with old method : 0.002467632293701172 time for calcul the mask position with numpy : 0.02387690544128418 nb_pixel_total : 51113 time to create 1 rle with old method : 0.05221271514892578 time for calcul the mask position with numpy : 0.024410009384155273 nb_pixel_total : 90263 time to create 1 rle with old method : 0.0933077335357666 time for calcul the mask position with numpy : 0.02359151840209961 nb_pixel_total : 71908 time to create 1 rle with old method : 0.09092020988464355 time for calcul the mask position with numpy : 0.024146556854248047 nb_pixel_total : 175556 time to create 1 rle with new method : 0.5119922161102295 time for calcul the mask position with numpy : 0.025084733963012695 nb_pixel_total : 14743 time to create 1 rle with old method : 0.01563096046447754 time for calcul the mask position with numpy : 0.02389836311340332 nb_pixel_total : 6410 time to create 1 rle with old method : 0.0067234039306640625 time for calcul the mask position with numpy : 0.023000240325927734 nb_pixel_total : 12939 time to create 1 rle with old method : 0.013631820678710938 time for calcul the mask position with numpy : 0.02283024787902832 nb_pixel_total : 7058 time to create 1 rle with old method : 0.007571697235107422 time for calcul the mask position with numpy : 0.023397207260131836 nb_pixel_total : 6781 time to create 1 rle with old method : 0.007455587387084961 time for calcul the mask position with numpy : 0.023805856704711914 nb_pixel_total : 34474 time to create 1 rle with old method : 0.03486037254333496 time for calcul the mask position with numpy : 0.024099111557006836 nb_pixel_total : 2795 time to create 1 rle with old method : 0.0030748844146728516 time for calcul the mask position with numpy : 0.02598881721496582 nb_pixel_total : 95579 time to create 1 rle with old method : 0.10009384155273438 time for calcul the mask position with numpy : 0.022491931915283203 nb_pixel_total : 7098 time to create 1 rle with old method : 0.007881879806518555 create new chi : 3.1320178508758545 time to delete rle : 0.021131038665771484 batch 1 Loaded 24 chid ids of type : 3726 Number RLEs to save : 14007 TO DO : save crop sub photo not yet done ! save time : 0.8886520862579346 nb_obj : 26 nb_hashtags : 7 time to prepare the origin masks : 9.227350950241089 time for calcul the mask position with numpy : 0.09885430335998535 nb_pixel_total : 5984204 time to create 1 rle with new method : 0.19100737571716309 time for calcul the mask position with numpy : 0.027692794799804688 nb_pixel_total : 2946 time to create 1 rle with old method : 0.003206968307495117 time for calcul the mask position with numpy : 0.028452634811401367 nb_pixel_total : 16137 time to create 1 rle with old method : 0.017365455627441406 time for calcul the mask position with numpy : 0.028785228729248047 nb_pixel_total : 8991 time to create 1 rle with old method : 0.010037660598754883 time for calcul the mask position with numpy : 0.028043270111083984 nb_pixel_total : 6225 time to create 1 rle with old method : 0.006994962692260742 time for calcul the mask position with numpy : 0.027720212936401367 nb_pixel_total : 18561 time to create 1 rle with old method : 0.019771099090576172 time for calcul the mask position with numpy : 0.02752685546875 nb_pixel_total : 3359 time to create 1 rle with old method : 0.0036725997924804688 time for calcul the mask position with numpy : 0.026906728744506836 nb_pixel_total : 1536 time to create 1 rle with old method : 0.0016837120056152344 time for calcul the mask position with numpy : 0.026981353759765625 nb_pixel_total : 1737 time to create 1 rle with old method : 0.0019066333770751953 time for calcul the mask position with numpy : 0.026929855346679688 nb_pixel_total : 15453 time to create 1 rle with old method : 0.016724348068237305 time for calcul the mask position with numpy : 0.02712249755859375 nb_pixel_total : 20152 time to create 1 rle with old method : 0.020963668823242188 time for calcul the mask position with numpy : 0.027042865753173828 nb_pixel_total : 344 time to create 1 rle with old method : 0.0005974769592285156 time for calcul the mask position with numpy : 0.02720952033996582 nb_pixel_total : 8517 time to create 1 rle with old method : 0.009371280670166016 time for calcul the mask position with numpy : 0.030436038970947266 nb_pixel_total : 163369 time to create 1 rle with new method : 0.15208172798156738 time for calcul the mask position with numpy : 0.028761625289916992 nb_pixel_total : 38718 time to create 1 rle with old method : 0.041049957275390625 time for calcul the mask position with numpy : 0.028700590133666992 nb_pixel_total : 17302 time to create 1 rle with old method : 0.019572734832763672 time for calcul the mask position with numpy : 0.030417680740356445 nb_pixel_total : 46217 time to create 1 rle with old method : 0.04957103729248047 time for calcul the mask position with numpy : 0.028990507125854492 nb_pixel_total : 8428 time to create 1 rle with old method : 0.011362791061401367 time for calcul the mask position with numpy : 0.031606435775756836 nb_pixel_total : 14321 time to create 1 rle with old method : 0.020003557205200195 time for calcul the mask position with numpy : 0.0335850715637207 nb_pixel_total : 65566 time to create 1 rle with old method : 0.07299065589904785 time for calcul the mask position with numpy : 0.028921127319335938 nb_pixel_total : 41346 time to create 1 rle with old method : 0.04438376426696777 time for calcul the mask position with numpy : 0.0288388729095459 nb_pixel_total : 70877 time to create 1 rle with old method : 0.07661199569702148 time for calcul the mask position with numpy : 0.030338287353515625 nb_pixel_total : 94708 time to create 1 rle with old method : 0.10498714447021484 time for calcul the mask position with numpy : 0.03182673454284668 nb_pixel_total : 352575 time to create 1 rle with new method : 0.14480161666870117 time for calcul the mask position with numpy : 0.029896259307861328 nb_pixel_total : 1535 time to create 1 rle with old method : 0.0018074512481689453 time for calcul the mask position with numpy : 0.028035402297973633 nb_pixel_total : 13157 time to create 1 rle with old method : 0.014324188232421875 time for calcul the mask position with numpy : 0.028428077697753906 nb_pixel_total : 33959 time to create 1 rle with old method : 0.03749394416809082 create new chi : 1.9950158596038818 time to delete rle : 0.001386880874633789 batch 1 Loaded 27 chid ids of type : 3726 Number RLEs to save : 12838 TO DO : save crop sub photo not yet done ! save time : 0.8990082740783691 nb_obj : 20 nb_hashtags : 6 time to prepare the origin masks : 9.410347938537598 time for calcul the mask position with numpy : 0.2605464458465576 nb_pixel_total : 6303090 time to create 1 rle with new method : 0.6258492469787598 time for calcul the mask position with numpy : 0.02014613151550293 nb_pixel_total : 67175 time to create 1 rle with old method : 0.07078123092651367 time for calcul the mask position with numpy : 0.02032947540283203 nb_pixel_total : 12230 time to create 1 rle with old method : 0.013574838638305664 time for calcul the mask position with numpy : 0.020073652267456055 nb_pixel_total : 4917 time to create 1 rle with old method : 0.005537271499633789 time for calcul the mask position with numpy : 0.02075505256652832 nb_pixel_total : 15291 time to create 1 rle with old method : 0.016568899154663086 time for calcul the mask position with numpy : 0.023476362228393555 nb_pixel_total : 379289 time to create 1 rle with new method : 0.25173139572143555 time for calcul the mask position with numpy : 0.020620346069335938 nb_pixel_total : 22355 time to create 1 rle with old method : 0.02372431755065918 time for calcul the mask position with numpy : 0.02023172378540039 nb_pixel_total : 23473 time to create 1 rle with old method : 0.025184154510498047 time for calcul the mask position with numpy : 0.02034759521484375 nb_pixel_total : 22861 time to create 1 rle with old method : 0.025038957595825195 time for calcul the mask position with numpy : 0.0211179256439209 nb_pixel_total : 45689 time to create 1 rle with old method : 0.049234628677368164 time for calcul the mask position with numpy : 0.02072739601135254 nb_pixel_total : 11087 time to create 1 rle with old method : 0.012618303298950195 time for calcul the mask position with numpy : 0.020700693130493164 nb_pixel_total : 3412 time to create 1 rle with old method : 0.0037577152252197266 time for calcul the mask position with numpy : 0.021131038665771484 nb_pixel_total : 15802 time to create 1 rle with old method : 0.01684093475341797 time for calcul the mask position with numpy : 0.020622968673706055 nb_pixel_total : 26422 time to create 1 rle with old method : 0.02846693992614746 time for calcul the mask position with numpy : 0.021022319793701172 nb_pixel_total : 10765 time to create 1 rle with old method : 0.012015104293823242 time for calcul the mask position with numpy : 0.021477937698364258 nb_pixel_total : 3979 time to create 1 rle with old method : 0.004575490951538086 time for calcul the mask position with numpy : 0.021219968795776367 nb_pixel_total : 15447 time to create 1 rle with old method : 0.01657581329345703 time for calcul the mask position with numpy : 0.019875049591064453 nb_pixel_total : 21289 time to create 1 rle with old method : 0.022707223892211914 time for calcul the mask position with numpy : 0.019418716430664062 nb_pixel_total : 2396 time to create 1 rle with old method : 0.0028085708618164062 time for calcul the mask position with numpy : 0.020671367645263672 nb_pixel_total : 13346 time to create 1 rle with old method : 0.01466822624206543 time for calcul the mask position with numpy : 0.020135164260864258 nb_pixel_total : 29925 time to create 1 rle with old method : 0.032000064849853516 create new chi : 1.993842363357544 time to delete rle : 0.0010466575622558594 batch 1 Loaded 21 chid ids of type : 3726 Number RLEs to save : 10896 TO DO : save crop sub photo not yet done ! save time : 0.8751931190490723 nb_obj : 21 nb_hashtags : 6 time to prepare the origin masks : 9.955316543579102 time for calcul the mask position with numpy : 0.5470066070556641 nb_pixel_total : 6508516 time to create 1 rle with new method : 0.7049863338470459 time for calcul the mask position with numpy : 0.02186417579650879 nb_pixel_total : 23403 time to create 1 rle with old method : 0.025662660598754883 time for calcul the mask position with numpy : 0.021067142486572266 nb_pixel_total : 86195 time to create 1 rle with old method : 0.09336304664611816 time for calcul the mask position with numpy : 0.020473003387451172 nb_pixel_total : 1000 time to create 1 rle with old method : 0.0013294219970703125 time for calcul the mask position with numpy : 0.020769119262695312 nb_pixel_total : 55626 time to create 1 rle with old method : 0.059938669204711914 time for calcul the mask position with numpy : 0.02485966682434082 nb_pixel_total : 877 time to create 1 rle with old method : 0.0010848045349121094 time for calcul the mask position with numpy : 0.02097487449645996 nb_pixel_total : 31751 time to create 1 rle with old method : 0.03455328941345215 time for calcul the mask position with numpy : 0.020818233489990234 nb_pixel_total : 13959 time to create 1 rle with old method : 0.015349626541137695 time for calcul the mask position with numpy : 0.0344388484954834 nb_pixel_total : 11378 time to create 1 rle with old method : 0.01240396499633789 time for calcul the mask position with numpy : 0.03420591354370117 nb_pixel_total : 4078 time to create 1 rle with old method : 0.004700660705566406 time for calcul the mask position with numpy : 0.03403830528259277 nb_pixel_total : 43615 time to create 1 rle with old method : 0.04571366310119629 time for calcul the mask position with numpy : 0.03544020652770996 nb_pixel_total : 2956 time to create 1 rle with old method : 0.0035147666931152344 time for calcul the mask position with numpy : 0.03614020347595215 nb_pixel_total : 82738 time to create 1 rle with old method : 0.09651708602905273 time for calcul the mask position with numpy : 0.03639578819274902 nb_pixel_total : 3589 time to create 1 rle with old method : 0.004193544387817383 time for calcul the mask position with numpy : 0.03544473648071289 nb_pixel_total : 1722 time to create 1 rle with old method : 0.002028942108154297 time for calcul the mask position with numpy : 0.03436470031738281 nb_pixel_total : 2689 time to create 1 rle with old method : 0.0030248165130615234 time for calcul the mask position with numpy : 0.03392815589904785 nb_pixel_total : 15568 time to create 1 rle with old method : 0.017615079879760742 time for calcul the mask position with numpy : 0.033288002014160156 nb_pixel_total : 25901 time to create 1 rle with old method : 0.02753734588623047 time for calcul the mask position with numpy : 0.03540968894958496 nb_pixel_total : 105849 time to create 1 rle with old method : 0.11359834671020508 time for calcul the mask position with numpy : 0.03391718864440918 nb_pixel_total : 5923 time to create 1 rle with old method : 0.006884574890136719 time for calcul the mask position with numpy : 0.0328061580657959 nb_pixel_total : 1115 time to create 1 rle with old method : 0.0012738704681396484 time for calcul the mask position with numpy : 0.0222017765045166 nb_pixel_total : 21792 time to create 1 rle with old method : 0.0242617130279541 create new chi : 2.5131497383117676 time to delete rle : 0.0011494159698486328 batch 1 Loaded 22 chid ids of type : 3726 Number RLEs to save : 10891 TO DO : save crop sub photo not yet done ! save time : 1.2332484722137451 nb_obj : 23 nb_hashtags : 6 time to prepare the origin masks : 10.791737079620361 time for calcul the mask position with numpy : 0.2742280960083008 nb_pixel_total : 5989774 time to create 1 rle with new method : 0.489849328994751 time for calcul the mask position with numpy : 0.020597457885742188 nb_pixel_total : 38361 time to create 1 rle with old method : 0.04058647155761719 time for calcul the mask position with numpy : 0.02015209197998047 nb_pixel_total : 4825 time to create 1 rle with old method : 0.005822181701660156 time for calcul the mask position with numpy : 0.020667552947998047 nb_pixel_total : 59096 time to create 1 rle with old method : 0.062052011489868164 time for calcul the mask position with numpy : 0.020167827606201172 nb_pixel_total : 17187 time to create 1 rle with old method : 0.018720626831054688 time for calcul the mask position with numpy : 0.028098106384277344 nb_pixel_total : 2253 time to create 1 rle with old method : 0.002454519271850586 time for calcul the mask position with numpy : 0.03367757797241211 nb_pixel_total : 61762 time to create 1 rle with old method : 0.06339216232299805 time for calcul the mask position with numpy : 0.03317427635192871 nb_pixel_total : 3844 time to create 1 rle with old method : 0.004045963287353516 time for calcul the mask position with numpy : 0.033933401107788086 nb_pixel_total : 10366 time to create 1 rle with old method : 0.012120962142944336 time for calcul the mask position with numpy : 0.0340878963470459 nb_pixel_total : 81730 time to create 1 rle with old method : 0.08762645721435547 time for calcul the mask position with numpy : 0.03399348258972168 nb_pixel_total : 269785 time to create 1 rle with new method : 0.3925788402557373 time for calcul the mask position with numpy : 0.03150486946105957 nb_pixel_total : 58363 time to create 1 rle with old method : 0.06251287460327148 time for calcul the mask position with numpy : 0.03516268730163574 nb_pixel_total : 77025 time to create 1 rle with old method : 0.08964681625366211 time for calcul the mask position with numpy : 0.03441119194030762 nb_pixel_total : 14092 time to create 1 rle with old method : 0.015298128128051758 time for calcul the mask position with numpy : 0.030890464782714844 nb_pixel_total : 69129 time to create 1 rle with old method : 0.10167312622070312 time for calcul the mask position with numpy : 0.020985841751098633 nb_pixel_total : 25219 time to create 1 rle with old method : 0.026643991470336914 time for calcul the mask position with numpy : 0.021164894104003906 nb_pixel_total : 47240 time to create 1 rle with old method : 0.0495600700378418 time for calcul the mask position with numpy : 0.021650075912475586 nb_pixel_total : 27158 time to create 1 rle with old method : 0.028329849243164062 time for calcul the mask position with numpy : 0.020770549774169922 nb_pixel_total : 119598 time to create 1 rle with old method : 0.15227651596069336 time for calcul the mask position with numpy : 0.021787405014038086 nb_pixel_total : 9713 time to create 1 rle with old method : 0.010689496994018555 time for calcul the mask position with numpy : 0.020714282989501953 nb_pixel_total : 3836 time to create 1 rle with old method : 0.004168987274169922 time for calcul the mask position with numpy : 0.020675182342529297 nb_pixel_total : 26498 time to create 1 rle with old method : 0.02791428565979004 time for calcul the mask position with numpy : 0.019918203353881836 nb_pixel_total : 29685 time to create 1 rle with old method : 0.03045797348022461 time for calcul the mask position with numpy : 0.020299434661865234 nb_pixel_total : 3701 time to create 1 rle with old method : 0.004084110260009766 create new chi : 2.7202913761138916 time to delete rle : 0.0020444393157958984 batch 1 Loaded 24 chid ids of type : 3726 Number RLEs to save : 15748 TO DO : save crop sub photo not yet done ! save time : 3.315882682800293 nb_obj : 12 nb_hashtags : 5 time to prepare the origin masks : 6.221591234207153 time for calcul the mask position with numpy : 0.29984068870544434 nb_pixel_total : 5710153 time to create 1 rle with new method : 0.6302473545074463 time for calcul the mask position with numpy : 0.020625829696655273 nb_pixel_total : 15985 time to create 1 rle with old method : 0.016690731048583984 time for calcul the mask position with numpy : 0.020856618881225586 nb_pixel_total : 35962 time to create 1 rle with old method : 0.04024696350097656 time for calcul the mask position with numpy : 0.0218503475189209 nb_pixel_total : 56848 time to create 1 rle with old method : 0.06126546859741211 time for calcul the mask position with numpy : 0.029883623123168945 nb_pixel_total : 25119 time to create 1 rle with old method : 0.027890920639038086 time for calcul the mask position with numpy : 0.022376537322998047 nb_pixel_total : 157271 time to create 1 rle with new method : 0.36966681480407715 time for calcul the mask position with numpy : 0.021034955978393555 nb_pixel_total : 6997 time to create 1 rle with old method : 0.008100509643554688 time for calcul the mask position with numpy : 0.020594358444213867 nb_pixel_total : 25778 time to create 1 rle with old method : 0.028087854385375977 time for calcul the mask position with numpy : 0.02093958854675293 nb_pixel_total : 179157 time to create 1 rle with new method : 0.2508432865142822 time for calcul the mask position with numpy : 0.025139331817626953 nb_pixel_total : 583568 time to create 1 rle with new method : 0.28008055686950684 time for calcul the mask position with numpy : 0.02198624610900879 nb_pixel_total : 115470 time to create 1 rle with old method : 0.12323594093322754 time for calcul the mask position with numpy : 0.02187323570251465 nb_pixel_total : 91093 time to create 1 rle with old method : 0.09721922874450684 time for calcul the mask position with numpy : 0.02297663688659668 nb_pixel_total : 46839 time to create 1 rle with old method : 0.05004572868347168 create new chi : 2.6398065090179443 time to delete rle : 0.0011703968048095703 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 13813 TO DO : save crop sub photo not yet done ! save time : 0.9160609245300293 nb_obj : 16 nb_hashtags : 5 time to prepare the origin masks : 6.249508857727051 time for calcul the mask position with numpy : 0.5371713638305664 nb_pixel_total : 6261069 time to create 1 rle with new method : 0.6137306690216064 time for calcul the mask position with numpy : 0.019652128219604492 nb_pixel_total : 15103 time to create 1 rle with old method : 0.015598058700561523 time for calcul the mask position with numpy : 0.020005226135253906 nb_pixel_total : 47133 time to create 1 rle with old method : 0.0499575138092041 time for calcul the mask position with numpy : 0.018796682357788086 nb_pixel_total : 7 time to create 1 rle with old method : 2.9325485229492188e-05 time for calcul the mask position with numpy : 0.019422054290771484 nb_pixel_total : 63580 time to create 1 rle with old method : 0.06563496589660645 time for calcul the mask position with numpy : 0.019741058349609375 nb_pixel_total : 58384 time to create 1 rle with old method : 0.06305694580078125 time for calcul the mask position with numpy : 0.02106189727783203 nb_pixel_total : 52501 time to create 1 rle with old method : 0.060064077377319336 time for calcul the mask position with numpy : 0.022277355194091797 nb_pixel_total : 170954 time to create 1 rle with new method : 0.48957085609436035 time for calcul the mask position with numpy : 0.021099567413330078 nb_pixel_total : 2073 time to create 1 rle with old method : 0.0022575855255126953 time for calcul the mask position with numpy : 0.020383596420288086 nb_pixel_total : 46442 time to create 1 rle with old method : 0.04891395568847656 time for calcul the mask position with numpy : 0.02029895782470703 nb_pixel_total : 36463 time to create 1 rle with old method : 0.03894758224487305 time for calcul the mask position with numpy : 0.02125382423400879 nb_pixel_total : 1541 time to create 1 rle with old method : 0.002006053924560547 time for calcul the mask position with numpy : 0.021801233291625977 nb_pixel_total : 89601 time to create 1 rle with old method : 0.12295699119567871 time for calcul the mask position with numpy : 0.021429777145385742 nb_pixel_total : 13695 time to create 1 rle with old method : 0.015056371688842773 time for calcul the mask position with numpy : 0.029233932495117188 nb_pixel_total : 3107 time to create 1 rle with old method : 0.0035941600799560547 time for calcul the mask position with numpy : 0.030844449996948242 nb_pixel_total : 41487 time to create 1 rle with old method : 0.04396486282348633 time for calcul the mask position with numpy : 0.031185388565063477 nb_pixel_total : 147100 time to create 1 rle with old method : 0.1527996063232422 create new chi : 2.7459027767181396 time to delete rle : 0.0009500980377197266 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 10298 TO DO : save crop sub photo not yet done ! save time : 0.9959752559661865 nb_obj : 29 nb_hashtags : 6 time to prepare the origin masks : 10.661030054092407 time for calcul the mask position with numpy : 0.07519721984863281 nb_pixel_total : 5346471 time to create 1 rle with new method : 0.15073037147521973 time for calcul the mask position with numpy : 0.028788089752197266 nb_pixel_total : 5400 time to create 1 rle with old method : 0.006714344024658203 time for calcul the mask position with numpy : 0.02985239028930664 nb_pixel_total : 191964 time to create 1 rle with new method : 0.14017820358276367 time for calcul the mask position with numpy : 0.02920675277709961 nb_pixel_total : 2123 time to create 1 rle with old method : 0.0025169849395751953 time for calcul the mask position with numpy : 0.0292971134185791 nb_pixel_total : 63774 time to create 1 rle with old method : 0.07213783264160156 time for calcul the mask position with numpy : 0.030208826065063477 nb_pixel_total : 3663 time to create 1 rle with old method : 0.0042841434478759766 time for calcul the mask position with numpy : 0.02936244010925293 nb_pixel_total : 70143 time to create 1 rle with old method : 0.07618999481201172 time for calcul the mask position with numpy : 0.028754472732543945 nb_pixel_total : 25071 time to create 1 rle with old method : 0.028016328811645508 time for calcul the mask position with numpy : 0.028856754302978516 nb_pixel_total : 23327 time to create 1 rle with old method : 0.0260927677154541 time for calcul the mask position with numpy : 0.030675411224365234 nb_pixel_total : 150161 time to create 1 rle with new method : 0.1449129581451416 time for calcul the mask position with numpy : 0.028963327407836914 nb_pixel_total : 6139 time to create 1 rle with old method : 0.007042407989501953 time for calcul the mask position with numpy : 0.029078960418701172 nb_pixel_total : 19010 time to create 1 rle with old method : 0.02134537696838379 time for calcul the mask position with numpy : 0.028853416442871094 nb_pixel_total : 6419 time to create 1 rle with old method : 0.007452964782714844 time for calcul the mask position with numpy : 0.02889704704284668 nb_pixel_total : 72284 time to create 1 rle with old method : 0.07886910438537598 time for calcul the mask position with numpy : 0.029106855392456055 nb_pixel_total : 65733 time to create 1 rle with old method : 0.07201766967773438 time for calcul the mask position with numpy : 0.0298922061920166 nb_pixel_total : 86775 time to create 1 rle with old method : 0.0948939323425293 time for calcul the mask position with numpy : 0.028858661651611328 nb_pixel_total : 41384 time to create 1 rle with old method : 0.044759511947631836 time for calcul the mask position with numpy : 0.02846217155456543 nb_pixel_total : 1097 time to create 1 rle with old method : 0.0014688968658447266 time for calcul the mask position with numpy : 0.02860426902770996 nb_pixel_total : 1587 time to create 1 rle with old method : 0.001873016357421875 time for calcul the mask position with numpy : 0.03184366226196289 nb_pixel_total : 12889 time to create 1 rle with old method : 0.014408111572265625 time for calcul the mask position with numpy : 0.02921319007873535 nb_pixel_total : 190049 time to create 1 rle with new method : 0.1315617561340332 time for calcul the mask position with numpy : 0.02930760383605957 nb_pixel_total : 82392 time to create 1 rle with old method : 0.09281539916992188 time for calcul the mask position with numpy : 0.03026890754699707 nb_pixel_total : 139083 time to create 1 rle with old method : 0.15085554122924805 time for calcul the mask position with numpy : 0.029010295867919922 nb_pixel_total : 27921 time to create 1 rle with old method : 0.03162097930908203 time for calcul the mask position with numpy : 0.028913259506225586 nb_pixel_total : 3368 time to create 1 rle with old method : 0.0041713714599609375 time for calcul the mask position with numpy : 0.029148578643798828 nb_pixel_total : 33647 time to create 1 rle with old method : 0.03670215606689453 time for calcul the mask position with numpy : 0.030562639236450195 nb_pixel_total : 323419 time to create 1 rle with new method : 0.12513279914855957 time for calcul the mask position with numpy : 0.028902053833007812 nb_pixel_total : 3608 time to create 1 rle with old method : 0.00420379638671875 time for calcul the mask position with numpy : 0.029103755950927734 nb_pixel_total : 43400 time to create 1 rle with old method : 0.047409772872924805 time for calcul the mask position with numpy : 0.02870655059814453 nb_pixel_total : 7939 time to create 1 rle with old method : 0.008977651596069336 create new chi : 2.6154885292053223 time to delete rle : 0.0016338825225830078 batch 1 Loaded 30 chid ids of type : 3726 Number RLEs to save : 17867 TO DO : save crop sub photo not yet done ! save time : 1.3359553813934326 nb_obj : 13 nb_hashtags : 5 time to prepare the origin masks : 10.5775306224823 time for calcul the mask position with numpy : 1.6549348831176758 nb_pixel_total : 6224876 time to create 1 rle with new method : 0.30095529556274414 time for calcul the mask position with numpy : 0.025727510452270508 nb_pixel_total : 44199 time to create 1 rle with old method : 0.04890155792236328 time for calcul the mask position with numpy : 0.03678011894226074 nb_pixel_total : 84213 time to create 1 rle with old method : 0.08866238594055176 time for calcul the mask position with numpy : 0.03152060508728027 nb_pixel_total : 2796 time to create 1 rle with old method : 0.0035517215728759766 time for calcul the mask position with numpy : 0.025255203247070312 nb_pixel_total : 28583 time to create 1 rle with old method : 0.030194997787475586 time for calcul the mask position with numpy : 0.021792888641357422 nb_pixel_total : 2528 time to create 1 rle with old method : 0.0028924942016601562 time for calcul the mask position with numpy : 0.021211862564086914 nb_pixel_total : 6343 time to create 1 rle with old method : 0.007183551788330078 time for calcul the mask position with numpy : 0.025843143463134766 nb_pixel_total : 23085 time to create 1 rle with old method : 0.027237415313720703 time for calcul the mask position with numpy : 0.028509140014648438 nb_pixel_total : 112089 time to create 1 rle with old method : 0.12284660339355469 time for calcul the mask position with numpy : 0.024364471435546875 nb_pixel_total : 2048 time to create 1 rle with old method : 0.002426624298095703 time for calcul the mask position with numpy : 0.02269744873046875 nb_pixel_total : 8129 time to create 1 rle with old method : 0.009205102920532227 time for calcul the mask position with numpy : 0.02253413200378418 nb_pixel_total : 86793 time to create 1 rle with old method : 0.09468626976013184 time for calcul the mask position with numpy : 0.02264714241027832 nb_pixel_total : 78269 time to create 1 rle with old method : 0.09434914588928223 time for calcul the mask position with numpy : 0.04035210609436035 nb_pixel_total : 346289 time to create 1 rle with new method : 0.31589794158935547 create new chi : 3.2087159156799316 time to delete rle : 0.00110626220703125 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 9901 TO DO : save crop sub photo not yet done ! save time : 0.8085536956787109 nb_obj : 14 nb_hashtags : 4 time to prepare the origin masks : 6.173696517944336 time for calcul the mask position with numpy : 0.17530393600463867 nb_pixel_total : 6701529 time to create 1 rle with new method : 0.42447352409362793 time for calcul the mask position with numpy : 0.028478622436523438 nb_pixel_total : 33074 time to create 1 rle with old method : 0.04664278030395508 time for calcul the mask position with numpy : 0.030211448669433594 nb_pixel_total : 18592 time to create 1 rle with old method : 0.02037334442138672 time for calcul the mask position with numpy : 0.04734182357788086 nb_pixel_total : 5754 time to create 1 rle with old method : 0.006540775299072266 time for calcul the mask position with numpy : 0.027339935302734375 nb_pixel_total : 5564 time to create 1 rle with old method : 0.008594274520874023 time for calcul the mask position with numpy : 0.027485370635986328 nb_pixel_total : 41255 time to create 1 rle with old method : 0.04937934875488281 time for calcul the mask position with numpy : 0.02537679672241211 nb_pixel_total : 11011 time to create 1 rle with old method : 0.01674509048461914 time for calcul the mask position with numpy : 0.030475616455078125 nb_pixel_total : 2489 time to create 1 rle with old method : 0.003286600112915039 time for calcul the mask position with numpy : 0.031195402145385742 nb_pixel_total : 11934 time to create 1 rle with old method : 0.014303922653198242 time for calcul the mask position with numpy : 0.029802560806274414 nb_pixel_total : 24893 time to create 1 rle with old method : 0.029120445251464844 time for calcul the mask position with numpy : 0.02736043930053711 nb_pixel_total : 7257 time to create 1 rle with old method : 0.008386611938476562 time for calcul the mask position with numpy : 0.04515790939331055 nb_pixel_total : 138783 time to create 1 rle with old method : 0.1658339500427246 time for calcul the mask position with numpy : 0.04075193405151367 nb_pixel_total : 3315 time to create 1 rle with old method : 0.003862142562866211 time for calcul the mask position with numpy : 0.05430865287780762 nb_pixel_total : 17018 time to create 1 rle with old method : 0.021111011505126953 time for calcul the mask position with numpy : 0.04562854766845703 nb_pixel_total : 27772 time to create 1 rle with old method : 0.030538558959960938 create new chi : 1.5504138469696045 time to delete rle : 0.0010635852813720703 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 7136 TO DO : save crop sub photo not yet done ! save time : 0.5996484756469727 map_output_result : {1335474142: (0.0, 'Should be the crop_list due to order', 0.0), 1335474138: (0.0, 'Should be the crop_list due to order', 0.0), 1335472989: (0.0, 'Should be the crop_list due to order', 0.0), 1335472873: (0.0, 'Should be the crop_list due to order', 0.0), 1335472785: (0.0, 'Should be the crop_list due to order', 0.0), 1335460491: (0.0, 'Should be the crop_list due to order', 0.0), 1335460487: (0.0, 'Should be the crop_list due to order', 0.0), 1335460482: (0.0, 'Should be the crop_list due to order', 0.0), 1335460478: (0.0, 'Should be the crop_list due to order', 0.0), 1335460424: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1335474142, 1335474138, 1335472989, 1335472873, 1335472785, 1335460491, 1335460487, 1335460482, 1335460478, 1335460424] Looping around the photos to save general results len do output : 10 /1335474142.Didn't retrieve data . /1335474138.Didn't retrieve data . /1335472989.Didn't retrieve data . /1335472873.Didn't retrieve data . /1335472785.Didn't retrieve data . /1335460491.Didn't retrieve data . /1335460487.Didn't retrieve data . /1335460482.Didn't retrieve data . /1335460478.Didn't retrieve data . /1335460424.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474142', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474138', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472989', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472873', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472785', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460491', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460487', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460482', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460478', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460424', None, None, None, None, None, '2567679') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014235258102416992 save_final save missing photos in datou_result : time spend for datou_step_exec : 128.12162375450134 time spend to save output : 0.03250932693481445 total time spend for step 5 : 128.15413308143616 step6:crop_condition Fri Feb 7 22:21:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 207 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738963263_1192779 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.446570634841919 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738963271_1192779 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.9154243469238281 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738963277_1192779 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.6211488246917725 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738963281_1192779 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.6138253211975098 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! About to upload 10 photos upload in portfolio : 4869462 init cache_photo without model_param we have 10 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738963288_1192779 we have uploaded 10 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.46749210357666 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738963293_1192779 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9160020351409912 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738963298_1192779 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.3668911457061768 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335474142, 1335474138, 1335472989, 1335472873, 1335472785, 1335460491, 1335460487, 1335460482, 1335460478, 1335460424] Looping around the photos to save general results len do output : 39 /1335619218Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619219Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619220Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619221Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619222Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619227Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619228Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619229Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619230Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619231Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619232Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619233Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619234Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619237Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619238Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619239Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619240Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619241Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619242Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619243Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619246Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619247Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619248Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619249Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619260Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619263Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619267Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619271Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619275Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619279Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619283Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619287Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619290Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619294Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619344Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619347Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619571Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619576Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335619580Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474142', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474138', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472989', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472873', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472785', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460491', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460487', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460482', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460478', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460424', None, None, None, None, None, '2567679') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 127 time used for this insertion : 0.11459636688232422 save_final save missing photos in datou_result : time spend for datou_step_exec : 39.585909605026245 time spend to save output : 0.11595821380615234 total time spend for step 6 : 39.7018678188324 step7:ventilate_hashtags_in_portfolio Fri Feb 7 22:21:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20351776 get user id for portfolio 20351776 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351776 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','metal','Carton_gris','Teint_Dans_La_Masse','Carton_brun','mal_croppe','environnement','papier','flou','cartonnette','autre_refus','plastique')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351776 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','metal','Carton_gris','Teint_Dans_La_Masse','Carton_brun','mal_croppe','environnement','papier','flou','cartonnette','autre_refus','plastique')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351776 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','metal','Carton_gris','Teint_Dans_La_Masse','Carton_brun','mal_croppe','environnement','papier','flou','cartonnette','autre_refus','plastique')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20351958,20351959,20351960,20351961,20351962,20351963,20351964,20351965,20351966,20351967,20351968,20351969?tags=kraft,metal,Carton_gris,Teint_Dans_La_Masse,Carton_brun,mal_croppe,environnement,papier,flou,cartonnette,autre_refus,plastique Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1335474142, 1335474138, 1335472989, 1335472873, 1335472785, 1335460491, 1335460487, 1335460482, 1335460478, 1335460424] Looping around the photos to save general results len do output : 1 /20351776. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474142', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474138', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472989', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472873', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472785', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460491', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460487', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460482', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460478', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460424', None, None, None, None, None, '2567679') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01979851722717285 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.090637445449829 time spend to save output : 0.02024555206298828 total time spend for step 7 : 2.1108829975128174 step8:final Fri Feb 7 22:21:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1335474142: ('0.09932132178283122',), 1335474138: ('0.09932132178283122',), 1335472989: ('0.09932132178283122',), 1335472873: ('0.09932132178283122',), 1335472785: ('0.09932132178283122',), 1335460491: ('0.09932132178283122',), 1335460487: ('0.09932132178283122',), 1335460482: ('0.09932132178283122',), 1335460478: ('0.09932132178283122',), 1335460424: ('0.09932132178283122',)} new output for save of step final : {1335474142: ('0.09932132178283122',), 1335474138: ('0.09932132178283122',), 1335472989: ('0.09932132178283122',), 1335472873: ('0.09932132178283122',), 1335472785: ('0.09932132178283122',), 1335460491: ('0.09932132178283122',), 1335460487: ('0.09932132178283122',), 1335460482: ('0.09932132178283122',), 1335460478: ('0.09932132178283122',), 1335460424: ('0.09932132178283122',)} [1335474142, 1335474138, 1335472989, 1335472873, 1335472785, 1335460491, 1335460487, 1335460482, 1335460478, 1335460424] Looping around the photos to save general results len do output : 10 /1335474142.Didn't retrieve data . /1335474138.Didn't retrieve data . /1335472989.Didn't retrieve data . /1335472873.Didn't retrieve data . /1335472785.Didn't retrieve data . /1335460491.Didn't retrieve data . /1335460487.Didn't retrieve data . /1335460482.Didn't retrieve data . /1335460478.Didn't retrieve data . /1335460424.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474142', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474138', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472989', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472873', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472785', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460491', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460487', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460482', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460478', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460424', None, None, None, None, None, '2567679') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.02232074737548828 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.12543296813964844 time spend to save output : 0.02321314811706543 total time spend for step 8 : 0.14864611625671387 step9:velours_tree Fri Feb 7 22:21:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.10714149475097656 time spend to save output : 9.250640869140625e-05 total time spend for step 9 : 0.10723400115966797 step10:send_mail_cod Fri Feb 7 22:21:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20351776_07-02-2025_22_21_42.pdf 20351958 change filename to text .change filename to text .change filename to text .imagette203519581738963302 20351959 imagette203519591738963302 20351963 imagette203519631738963302 20351966 imagette203519661738963302 20351967 change filename to text .change filename to text .change filename to text .change filename to text .imagette203519671738963302 20351968 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203519681738963303 20351969 change filename to text .change filename to text .imagette203519691738963304 20351960 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203519601738963304 20351961 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203519611738963304 20351962 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203519621738963305 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20351776 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20351958,20351959,20351960,20351961,20351962,20351963,20351964,20351965,20351966,20351967,20351968,20351969?tags=kraft,metal,Carton_gris,Teint_Dans_La_Masse,Carton_brun,mal_croppe,environnement,papier,flou,cartonnette,autre_refus,plastique your option no_mail is active, we will not send the real mail to your client args[1335474142] : ('0.09932132178283122',) no score found for photo 1335474142 apple ('0.09932132178283122',) We are sending mail with results at report@fotonower.com args[1335474138] : ('0.09932132178283122',) no score found for photo 1335474138 apple ('0.09932132178283122',) We are sending mail with results at report@fotonower.com args[1335472989] : ('0.09932132178283122',) no score found for photo 1335472989 apple ('0.09932132178283122',) We are sending mail with results at report@fotonower.com args[1335472873] : ('0.09932132178283122',) no score found for photo 1335472873 apple ('0.09932132178283122',) We are sending mail with results at report@fotonower.com args[1335472785] : ('0.09932132178283122',) no score found for photo 1335472785 apple ('0.09932132178283122',) We are sending mail with results at report@fotonower.com args[1335460491] : ('0.09932132178283122',) no score found for photo 1335460491 apple ('0.09932132178283122',) We are sending mail with results at report@fotonower.com args[1335460487] : ('0.09932132178283122',) no score found for photo 1335460487 apple ('0.09932132178283122',) We are sending mail with results at report@fotonower.com args[1335460482] : ('0.09932132178283122',) no score found for photo 1335460482 apple ('0.09932132178283122',) We are sending mail with results at report@fotonower.com args[1335460478] : ('0.09932132178283122',) no score found for photo 1335460478 apple ('0.09932132178283122',) We are sending mail with results at report@fotonower.com args[1335460424] : ('0.09932132178283122',) no score found for photo 1335460424 apple ('0.09932132178283122',) We are sending mail with results at report@fotonower.com refus_total : 0.09932132178283122 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20351776 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1335460398,1335460401,1335460414,1335460424,1335460478,1335460482,1335460487,1335460491,1335472785,1335472873,1335474138,1335474142,1335460418,1335472989) Found this number of photos: 14 begin to download photo : 1335460398 begin to download photo : 1335460424 begin to download photo : 1335460487 begin to download photo : 1335472873 begin to download photo : 1335460418 download finish for photo 1335460424 begin to download photo : 1335460478 download finish for photo 1335472873 begin to download photo : 1335474138 download finish for photo 1335460418 begin to download photo : 1335472989 download finish for photo 1335460398 begin to download photo : 1335460401 download finish for photo 1335460487 begin to download photo : 1335460491 download finish for photo 1335474138 begin to download photo : 1335474142 download finish for photo 1335460478 begin to download photo : 1335460482 download finish for photo 1335472989 download finish for photo 1335460491 begin to download photo : 1335472785 download finish for photo 1335460401 begin to download photo : 1335460414 download finish for photo 1335474142 download finish for photo 1335460482 download finish for photo 1335472785 download finish for photo 1335460414 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351776_07-02-2025_22_21_42.pdf results_Auto_P20351776_07-02-2025_22_21_42.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351776_07-02-2025_22_21_42.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20351776','results_Auto_P20351776_07-02-2025_22_21_42.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351776_07-02-2025_22_21_42.pdf','pdf','','0.75','0.09932132178283122') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1335474142, 1335474138, 1335472989, 1335472873, 1335472785, 1335460491, 1335460487, 1335460482, 1335460478, 1335460424] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474142', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474138', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472989', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472873', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472785', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460491', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460487', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460482', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460478', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460424', None, None, None, None, None, '2567679') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.019553422927856445 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.4474873542785645 time spend to save output : 0.019901752471923828 total time spend for step 10 : 6.467389106750488 step11:split_time_score Fri Feb 7 22:21:49 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('09', 14),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 07022025 20351776 Nombre de photos uploadées : 14 / 23040 (0%) 07022025 20351776 Nombre de photos taguées (types de déchets): 0 / 14 (0%) 07022025 20351776 Nombre de photos taguées (volume) : 0 / 14 (0%) elapsed_time : load_data_split_time_score 3.337860107421875e-06 elapsed_time : order_list_meta_photo_and_scores 7.867813110351562e-06 ?????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0012669563293457031 elapsed_time : insert_dashboard_record_day_entry 0.14038991928100586 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351770 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351771 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351772 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351773 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351774 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351775 order by id desc limit 1 Qualite : 0.09932132178283122 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351776_07-02-2025_22_21_42.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351776 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351776 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351777 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351778 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351779 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351782 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351807 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351814 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351815 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351816 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351817 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351818 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351819 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351820 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351821 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351822 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351841 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351842 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351861 order by id desc limit 1 Qualite : 0.1061343927720056 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351862_07-02-2025_22_19_35.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351862 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351862 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351863 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351864 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351865 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351866 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'07022025': {'nb_upload': 14, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1335474142, 1335474138, 1335472989, 1335472873, 1335472785, 1335460491, 1335460487, 1335460482, 1335460478, 1335460424] Looping around the photos to save general results len do output : 1 /20351776Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474142', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335474138', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472989', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472873', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335472785', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460491', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460487', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460482', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460478', None, None, None, None, None, '2567679') ('3459', None, None, None, None, None, None, None, '2567679') ('3459', '20351776', '1335460424', None, None, None, None, None, '2567679') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013730525970458984 save_final save missing photos in datou_result : time spend for datou_step_exec : 22.638904333114624 time spend to save output : 0.014119148254394531 total time spend for step 11 : 22.65302348136902 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 187.78user 121.68system 5:48.67elapsed 88%CPU (0avgtext+0avgdata 4983408maxresident)k 264624inputs+135032outputs (10794major+13284221minor)pagefaults 0swaps