python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 341699 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 28>10 please execute split_portfolio.py -i 20799180 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2612418'] with mtr_portfolio_ids : ['20799180'] and first list_photo_ids : [] new path : /proc/341699/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.406723976135254 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue Feb 25 10:42:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 1713 wait 20 seconds l 3637 free memory gpu now : 1713 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-25 10:42:53.020109: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-25 10:42:53.047157: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-25 10:42:53.048795: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fb35c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-25 10:42:53.048848: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-25 10:42:53.051456: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-25 10:42:53.282287: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x21bfec60 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-25 10:42:53.282332: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-25 10:42:53.283312: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-25 10:42:53.283724: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-25 10:42:53.287161: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-25 10:42:53.290299: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-25 10:42:53.290823: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-25 10:42:53.293585: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-25 10:42:53.294648: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-25 10:42:53.299331: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-25 10:42:53.300249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-25 10:42:53.300324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-25 10:42:53.300819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-25 10:42:53.300833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-25 10:42:53.300842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-25 10:42:53.301593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1336 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-25 10:42:53.627361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-25 10:42:53.627485: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-25 10:42:53.627503: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-25 10:42:53.627519: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-25 10:42:53.627534: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-25 10:42:53.627549: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-25 10:42:53.627565: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-25 10:42:53.627579: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-25 10:42:53.628273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-25 10:42:53.629103: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-25 10:42:53.629131: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-25 10:42:53.629145: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-25 10:42:53.629158: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-25 10:42:53.629170: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-25 10:42:53.629183: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-25 10:42:53.629195: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-25 10:42:53.629208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-25 10:42:53.629857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-25 10:42:53.629890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-25 10:42:53.629898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-25 10:42:53.629905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-25 10:42:53.630605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1336 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-25 10:43:01.655414: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-25 10:43:01.839844: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-25 10:43:03.156765: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-25 10:43:03.156842: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-25 10:43:03.163253: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-25 10:43:03.163273: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-25 10:43:03.213557: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-25 10:43:03.213627: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-25 10:43:03.253967: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-25 10:43:03.254000: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-25 10:43:03.298635: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-25 10:43:03.298661: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-25 10:43:03.325560: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-02-25 10:43:03.343668: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.344111: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 743.29M (779393536 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.344553: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 668.96M (701454336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.344989: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 602.06M (631309056 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.345445: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.346934: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.347391: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.354016: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.354472: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.360099: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.360555: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.361993: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.362448: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.389504: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.389969: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.390424: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.390876: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.394339: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.394795: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.410671: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.411149: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.411604: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.412070: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.425102: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.425562: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.426017: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.426471: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.434806: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.435283: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.447895: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.448354: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.452471: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.452928: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.453584: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.454094: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.464991: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.465452: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.465923: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.466380: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.466836: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.467311: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.536693: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.537234: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.544857: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.545321: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.560018: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.560675: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.561308: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.561927: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.565882: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.566394: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.566870: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.567340: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.568200: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.568215: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-02-25 10:43:03.578244: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.578712: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.587374: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.587842: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.588314: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.588777: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.589248: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-25 10:43:03.589711: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 825.88M (865992704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 Detection mask done ! Trying to reset tf kernel 343237 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 520 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1713 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0002579689025878906 nb_pixel_total : 8275 time to create 1 rle with old method : 0.009970903396606445 length of segment : 228 time for calcul the mask position with numpy : 0.0003261566162109375 nb_pixel_total : 14860 time to create 1 rle with old method : 0.020193815231323242 length of segment : 160 time for calcul the mask position with numpy : 0.01124429702758789 nb_pixel_total : 610423 time to create 1 rle with new method : 0.05032634735107422 length of segment : 1341 time for calcul the mask position with numpy : 0.0004935264587402344 nb_pixel_total : 28951 time to create 1 rle with old method : 0.034326791763305664 length of segment : 248 time for calcul the mask position with numpy : 0.0006468296051025391 nb_pixel_total : 51431 time to create 1 rle with old method : 0.06078529357910156 length of segment : 254 time for calcul the mask position with numpy : 0.00046706199645996094 nb_pixel_total : 29240 time to create 1 rle with old method : 0.036859989166259766 length of segment : 225 time for calcul the mask position with numpy : 0.0007650852203369141 nb_pixel_total : 53620 time to create 1 rle with old method : 0.06167101860046387 length of segment : 298 time for calcul the mask position with numpy : 0.0009210109710693359 nb_pixel_total : 16797 time to create 1 rle with old method : 0.027517080307006836 length of segment : 299 time for calcul the mask position with numpy : 0.013434410095214844 nb_pixel_total : 733232 time to create 1 rle with new method : 0.17449235916137695 length of segment : 1255 time for calcul the mask position with numpy : 0.003307819366455078 nb_pixel_total : 221951 time to create 1 rle with new method : 0.009181022644042969 length of segment : 895 time for calcul the mask position with numpy : 0.0034723281860351562 nb_pixel_total : 315175 time to create 1 rle with new method : 0.007549285888671875 length of segment : 934 time for calcul the mask position with numpy : 7.462501525878906e-05 nb_pixel_total : 1109 time to create 1 rle with old method : 0.0015175342559814453 length of segment : 38 time for calcul the mask position with numpy : 0.0013380050659179688 nb_pixel_total : 95592 time to create 1 rle with old method : 0.11252737045288086 length of segment : 638 time for calcul the mask position with numpy : 0.0013620853424072266 nb_pixel_total : 100067 time to create 1 rle with old method : 0.12057900428771973 length of segment : 560 time for calcul the mask position with numpy : 0.002745389938354492 nb_pixel_total : 123801 time to create 1 rle with old method : 0.13902950286865234 length of segment : 296 time for calcul the mask position with numpy : 0.00305938720703125 nb_pixel_total : 174146 time to create 1 rle with new method : 0.007136106491088867 length of segment : 664 time for calcul the mask position with numpy : 0.004014492034912109 nb_pixel_total : 212210 time to create 1 rle with new method : 0.01247262954711914 length of segment : 830 time for calcul the mask position with numpy : 9.202957153320312e-05 nb_pixel_total : 647 time to create 1 rle with old method : 0.0012319087982177734 length of segment : 36 time for calcul the mask position with numpy : 0.000400543212890625 nb_pixel_total : 7223 time to create 1 rle with old method : 0.012022733688354492 length of segment : 151 time for calcul the mask position with numpy : 0.016707181930541992 nb_pixel_total : 856736 time to create 1 rle with new method : 0.20297527313232422 length of segment : 1358 time for calcul the mask position with numpy : 0.00021028518676757812 nb_pixel_total : 12714 time to create 1 rle with old method : 0.015588760375976562 length of segment : 202 time for calcul the mask position with numpy : 0.0007617473602294922 nb_pixel_total : 28242 time to create 1 rle with old method : 0.03399491310119629 length of segment : 380 time for calcul the mask position with numpy : 0.0001919269561767578 nb_pixel_total : 12393 time to create 1 rle with old method : 0.015349626541137695 length of segment : 117 time for calcul the mask position with numpy : 0.0008294582366943359 nb_pixel_total : 55077 time to create 1 rle with old method : 0.06515622138977051 length of segment : 395 time for calcul the mask position with numpy : 0.003299236297607422 nb_pixel_total : 294113 time to create 1 rle with new method : 0.007642269134521484 length of segment : 1001 time for calcul the mask position with numpy : 0.008059263229370117 nb_pixel_total : 389177 time to create 1 rle with new method : 0.03491640090942383 length of segment : 1821 time for calcul the mask position with numpy : 0.0003371238708496094 nb_pixel_total : 17395 time to create 1 rle with old method : 0.02160334587097168 length of segment : 157 time for calcul the mask position with numpy : 0.0004444122314453125 nb_pixel_total : 17456 time to create 1 rle with old method : 0.02231001853942871 length of segment : 168 time for calcul the mask position with numpy : 5.173683166503906e-05 nb_pixel_total : 801 time to create 1 rle with old method : 0.001077890396118164 length of segment : 50 time for calcul the mask position with numpy : 0.0013949871063232422 nb_pixel_total : 88242 time to create 1 rle with old method : 0.10653400421142578 length of segment : 441 time for calcul the mask position with numpy : 0.006615877151489258 nb_pixel_total : 368316 time to create 1 rle with new method : 0.015140295028686523 length of segment : 990 time for calcul the mask position with numpy : 0.0003457069396972656 nb_pixel_total : 17400 time to create 1 rle with old method : 0.02258753776550293 length of segment : 71 time for calcul the mask position with numpy : 0.0005309581756591797 nb_pixel_total : 25209 time to create 1 rle with old method : 0.032141923904418945 length of segment : 190 time for calcul the mask position with numpy : 0.007375001907348633 nb_pixel_total : 411790 time to create 1 rle with new method : 0.23005199432373047 length of segment : 1785 time for calcul the mask position with numpy : 0.0014672279357910156 nb_pixel_total : 79376 time to create 1 rle with old method : 0.08779549598693848 length of segment : 316 time for calcul the mask position with numpy : 0.00011086463928222656 nb_pixel_total : 824 time to create 1 rle with old method : 0.00104522705078125 length of segment : 54 time for calcul the mask position with numpy : 0.0008788108825683594 nb_pixel_total : 35306 time to create 1 rle with old method : 0.045769453048706055 length of segment : 299 time for calcul the mask position with numpy : 0.0033388137817382812 nb_pixel_total : 187205 time to create 1 rle with new method : 0.006542205810546875 length of segment : 780 time for calcul the mask position with numpy : 0.0001785755157470703 nb_pixel_total : 750 time to create 1 rle with old method : 0.0012385845184326172 length of segment : 67 time for calcul the mask position with numpy : 0.006523609161376953 nb_pixel_total : 445112 time to create 1 rle with new method : 0.031179428100585938 length of segment : 1357 time for calcul the mask position with numpy : 0.0021445751190185547 nb_pixel_total : 127659 time to create 1 rle with old method : 0.14410924911499023 length of segment : 588 time for calcul the mask position with numpy : 0.013034820556640625 nb_pixel_total : 1073434 time to create 1 rle with new method : 0.05565595626831055 length of segment : 1398 time for calcul the mask position with numpy : 0.005709409713745117 nb_pixel_total : 381983 time to create 1 rle with new method : 0.014008522033691406 length of segment : 1708 time for calcul the mask position with numpy : 0.0038061141967773438 nb_pixel_total : 175270 time to create 1 rle with new method : 0.0077686309814453125 length of segment : 949 time for calcul the mask position with numpy : 0.00022864341735839844 nb_pixel_total : 4088 time to create 1 rle with old method : 0.005128622055053711 length of segment : 92 time for calcul the mask position with numpy : 0.012653112411499023 nb_pixel_total : 761257 time to create 1 rle with new method : 0.10706686973571777 length of segment : 1367 time for calcul the mask position with numpy : 0.004492044448852539 nb_pixel_total : 366255 time to create 1 rle with new method : 0.012104511260986328 length of segment : 1818 time spent for convertir_results : 6.530057430267334 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 47 chid ids of type : 3663 Number RLEs to save : 29269 save missing photos in datou_result : time spend for datou_step_exec : 47.80301260948181 time spend to save output : 1.899688959121704 total time spend for step 1 : 49.702701568603516 step2:crop_condition Tue Feb 25 10:43:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 47 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1339522520, 1339522480, 1339522479, 1339522440, 1339522439, 1339522332, 1339522331, 1339522295, 1339522294, 1339522262] Looping around the photos to save general results len do output : 16 /-3686312674Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312677Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312676Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312683Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312693Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312692Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312662Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312659Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312690Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312668Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312678Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312679Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312687Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312700Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312660Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3686312661Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522520', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522480', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522479', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522440', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522439', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522332', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522331', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522295', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522294', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522262', None, None, None, None, None, '2612418') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 58 time used for this insertion : 0.02164316177368164 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.723449230194092 time spend to save output : 0.022491455078125 total time spend for step 2 : 4.745940685272217 step3:thcl Tue Feb 25 10:43:24 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.010131120681762695 time to convert the images to numpy array : 2.384185791015625e-06 time to import caffe and check if the image exist : 0.010325908660888672 time to convert the images to numpy array : 1.9073486328125e-06 time to import caffe and check if the image exist : 0.00850534439086914 time to convert the images to numpy array : 0.009139299392700195 time to import caffe and check if the image exist : 0.006695985794067383 time to convert the images to numpy array : 0.01414632797241211 time to import caffe and check if the image exist : 0.008427858352661133 time to convert the images to numpy array : 0.014986991882324219 time to import caffe and check if the image exist : 0.010332345962524414 time to convert the images to numpy array : 0.018527746200561523 time to import caffe and check if the image exist : 0.009942054748535156 time to convert the images to numpy array : 0.019313335418701172 time to import caffe and check if the image exist : 0.010657787322998047 time to convert the images to numpy array : 0.01866888999938965 time to import caffe and check if the image exist : 0.0069539546966552734 time to convert the images to numpy array : 0.20187759399414062 time to import caffe and check if the image exist : 0.008564949035644531 time to convert the images to numpy array : 0.21292352676391602 total time to convert the images to numpy array : 0.4583466053009033 list photo_ids error: [] list photo_ids correct : [-3686312687, -3686312700, -3686312676, -3686312683, -3686312674, -3686312677, -3686312662, -3686312659, -3686312678, -3686312679, -3686312660, -3686312661, -3686312690, -3686312668, -3686312693, -3686312692] number of photos to traite : 16 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1713 wait 20 seconds l 3637 free memory gpu now : 1713 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1494 wait 20 seconds l 3637 free memory gpu now : 1494 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.14363431930541992 time used to do the prediction : 0.2138984203338623 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1492 wait 20 seconds l 3637 free memory gpu now : 1492 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1492 wait 20 seconds l 3637 free memory gpu now : 1492 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.21063661575317383 time used to do the prediction : 0.16503500938415527 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.0967254638671875e-05 save missing photos in datou_result : time spend for datou_step_exec : 88.36291217803955 time spend to save output : 0.0003578662872314453 total time spend for step 3 : 88.36327004432678 step4:merge_mask_thcl_custom Tue Feb 25 10:44:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 47 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1339522520, 1339522480, 1339522479, 1339522440, 1339522439, 1339522332, 1339522331, 1339522295, 1339522294, 1339522262] Looping around the photos to save general results len do output : 10 /1339522520Didn't retrieve data .Didn't retrieve data . /1339522480Didn't retrieve data .Didn't retrieve data . /1339522479Didn't retrieve data .Didn't retrieve data . /1339522440Didn't retrieve data .Didn't retrieve data . /1339522439Didn't retrieve data .Didn't retrieve data . /1339522332Didn't retrieve data .Didn't retrieve data . /1339522331Didn't retrieve data .Didn't retrieve data . /1339522295Didn't retrieve data .Didn't retrieve data . /1339522294Didn't retrieve data .Didn't retrieve data . /1339522262Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522520', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522480', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522479', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522440', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522439', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522332', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522331', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522295', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522294', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522262', None, None, None, None, None, '2612418') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.03854966163635254 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.05852222442626953 time spend to save output : 0.038892269134521484 total time spend for step 4 : 0.09741449356079102 step5:rle_unique_nms_with_priority Tue Feb 25 10:44:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 8 nb_hashtags : 4 time to prepare the origin masks : 2.2855396270751953 time for calcul the mask position with numpy : 0.22617292404174805 nb_pixel_total : 1313353 time to create 1 rle with new method : 0.10140776634216309 time for calcul the mask position with numpy : 0.007187843322753906 nb_pixel_total : 3563 time to create 1 rle with old method : 0.0044248104095458984 time for calcul the mask position with numpy : 0.006420612335205078 nb_pixel_total : 758 time to create 1 rle with old method : 0.0010862350463867188 time for calcul the mask position with numpy : 0.0066432952880859375 nb_pixel_total : 51272 time to create 1 rle with old method : 0.056513071060180664 time for calcul the mask position with numpy : 0.006669759750366211 nb_pixel_total : 29187 time to create 1 rle with old method : 0.03300929069519043 time for calcul the mask position with numpy : 0.006574153900146484 nb_pixel_total : 16737 time to create 1 rle with old method : 0.018608808517456055 time for calcul the mask position with numpy : 0.010584831237792969 nb_pixel_total : 635698 time to create 1 rle with new method : 0.2646968364715576 time for calcul the mask position with numpy : 0.006523609161376953 nb_pixel_total : 14775 time to create 1 rle with old method : 0.016978025436401367 time for calcul the mask position with numpy : 0.006440162658691406 nb_pixel_total : 8257 time to create 1 rle with old method : 0.009679794311523438 create new chi : 0.8067417144775391 time to delete rle : 0.019202232360839844 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 5772 TO DO : save crop sub photo not yet done ! save time : 0.35652995109558105 nb_obj : 6 nb_hashtags : 2 time to prepare the origin masks : 0.7201199531555176 time for calcul the mask position with numpy : 0.038279056549072266 nb_pixel_total : 880653 time to create 1 rle with new method : 0.11020731925964355 time for calcul the mask position with numpy : 0.006577730178833008 nb_pixel_total : 29047 time to create 1 rle with old method : 0.03441977500915527 time for calcul the mask position with numpy : 0.006591081619262695 nb_pixel_total : 9429 time to create 1 rle with old method : 0.011188983917236328 time for calcul the mask position with numpy : 0.0070269107818603516 nb_pixel_total : 117716 time to create 1 rle with old method : 0.134202241897583 time for calcul the mask position with numpy : 0.0072858333587646484 nb_pixel_total : 203776 time to create 1 rle with new method : 0.029358863830566406 time for calcul the mask position with numpy : 0.015478849411010742 nb_pixel_total : 732536 time to create 1 rle with new method : 0.028943538665771484 time for calcul the mask position with numpy : 0.008063077926635742 nb_pixel_total : 100443 time to create 1 rle with old method : 0.11097121238708496 create new chi : 0.5497932434082031 time to delete rle : 0.0007278919219970703 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 8202 TO DO : save crop sub photo not yet done ! save time : 0.5568904876708984 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.2381303310394287 time for calcul the mask position with numpy : 0.0227663516998291 nb_pixel_total : 1687502 time to create 1 rle with new method : 0.03472399711608887 time for calcul the mask position with numpy : 0.006496906280517578 nb_pixel_total : 636 time to create 1 rle with old method : 0.0008213520050048828 time for calcul the mask position with numpy : 0.008124113082885742 nb_pixel_total : 211532 time to create 1 rle with new method : 0.029680490493774414 time for calcul the mask position with numpy : 0.007334232330322266 nb_pixel_total : 173930 time to create 1 rle with new method : 0.029031038284301758 create new chi : 0.13965511322021484 time to delete rle : 0.0004341602325439453 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 4152 TO DO : save crop sub photo not yet done ! save time : 0.29445338249206543 nb_obj : 7 nb_hashtags : 5 time to prepare the origin masks : 0.9041523933410645 time for calcul the mask position with numpy : 0.04769253730773926 nb_pixel_total : 1019892 time to create 1 rle with new method : 0.12180423736572266 time for calcul the mask position with numpy : 0.0063686370849609375 nb_pixel_total : 176 time to create 1 rle with old method : 0.00046563148498535156 time for calcul the mask position with numpy : 0.006875038146972656 nb_pixel_total : 74556 time to create 1 rle with old method : 0.08301901817321777 time for calcul the mask position with numpy : 0.006780147552490234 nb_pixel_total : 12685 time to create 1 rle with old method : 0.013734579086303711 time for calcul the mask position with numpy : 0.00674748420715332 nb_pixel_total : 49040 time to create 1 rle with old method : 0.05571317672729492 time for calcul the mask position with numpy : 0.00626063346862793 nb_pixel_total : 11723 time to create 1 rle with old method : 0.012609004974365234 time for calcul the mask position with numpy : 0.012236833572387695 nb_pixel_total : 898316 time to create 1 rle with new method : 0.19595742225646973 time for calcul the mask position with numpy : 0.006330013275146484 nb_pixel_total : 7212 time to create 1 rle with old method : 0.008063793182373047 create new chi : 0.6067874431610107 time to delete rle : 0.0007529258728027344 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 5687 TO DO : save crop sub photo not yet done ! save time : 0.3770885467529297 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.30100369453430176 time for calcul the mask position with numpy : 0.019095420837402344 nb_pixel_total : 1669142 time to create 1 rle with new method : 0.03185844421386719 time for calcul the mask position with numpy : 0.006636619567871094 nb_pixel_total : 798 time to create 1 rle with old method : 0.0011401176452636719 time for calcul the mask position with numpy : 0.007557868957519531 nb_pixel_total : 730 time to create 1 rle with old method : 0.0012552738189697266 time for calcul the mask position with numpy : 0.00872659683227539 nb_pixel_total : 385597 time to create 1 rle with new method : 0.030473709106445312 time for calcul the mask position with numpy : 0.006335735321044922 nb_pixel_total : 17333 time to create 1 rle with old method : 0.019262313842773438 create new chi : 0.13312888145446777 time to delete rle : 0.0004940032958984375 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 5322 TO DO : save crop sub photo not yet done ! save time : 0.35715770721435547 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 0.6453080177307129 time for calcul the mask position with numpy : 0.04860043525695801 nb_pixel_total : 1256540 time to create 1 rle with new method : 0.4415445327758789 time for calcul the mask position with numpy : 0.007018566131591797 nb_pixel_total : 17357 time to create 1 rle with old method : 0.019379615783691406 time for calcul the mask position with numpy : 0.009031057357788086 nb_pixel_total : 314822 time to create 1 rle with new method : 0.2058124542236328 time for calcul the mask position with numpy : 0.006294965744018555 nb_pixel_total : 25114 time to create 1 rle with old method : 0.02686905860900879 time for calcul the mask position with numpy : 0.008768558502197266 nb_pixel_total : 380719 time to create 1 rle with new method : 0.08017992973327637 time for calcul the mask position with numpy : 0.007210493087768555 nb_pixel_total : 79048 time to create 1 rle with old method : 0.08387637138366699 create new chi : 0.9639739990234375 time to delete rle : 0.0006666183471679688 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6969 TO DO : save crop sub photo not yet done ! save time : 0.42738938331604004 nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 0.20768189430236816 time for calcul the mask position with numpy : 0.021106243133544922 nb_pixel_total : 1850610 time to create 1 rle with new method : 0.09885549545288086 time for calcul the mask position with numpy : 0.008758544921875 nb_pixel_total : 186999 time to create 1 rle with new method : 0.179548978805542 time for calcul the mask position with numpy : 0.007261037826538086 nb_pixel_total : 820 time to create 1 rle with old method : 0.000993490219116211 time for calcul the mask position with numpy : 0.007749080657958984 nb_pixel_total : 35171 time to create 1 rle with old method : 0.03927302360534668 create new chi : 0.37241220474243164 time to delete rle : 0.0004239082336425781 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 3344 TO DO : save crop sub photo not yet done ! save time : 0.2429523468017578 nb_obj : 1 nb_hashtags : 1 time to prepare the origin masks : 0.2532005310058594 time for calcul the mask position with numpy : 0.059163570404052734 nb_pixel_total : 1571307 time to create 1 rle with new method : 0.14367103576660156 time for calcul the mask position with numpy : 0.012182950973510742 nb_pixel_total : 502293 time to create 1 rle with new method : 0.0896456241607666 create new chi : 0.31472253799438477 time to delete rle : 0.0003604888916015625 batch 1 Loaded 2 chid ids of type : 3726 Number RLEs to save : 3156 TO DO : save crop sub photo not yet done ! save time : 0.23189687728881836 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.8664605617523193 time for calcul the mask position with numpy : 0.037566423416137695 nb_pixel_total : 829674 time to create 1 rle with new method : 0.26786375045776367 time for calcul the mask position with numpy : 0.008681774139404297 nb_pixel_total : 136029 time to create 1 rle with old method : 0.16196107864379883 time for calcul the mask position with numpy : 0.015573978424072266 nb_pixel_total : 980512 time to create 1 rle with new method : 0.1886591911315918 time for calcul the mask position with numpy : 0.007837295532226562 nb_pixel_total : 127385 time to create 1 rle with old method : 0.14179778099060059 create new chi : 0.8483662605285645 time to delete rle : 0.0006802082061767578 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 5902 TO DO : save crop sub photo not yet done ! save time : 0.37721896171569824 nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.5756864547729492 time for calcul the mask position with numpy : 0.03430318832397461 nb_pixel_total : 1046882 time to create 1 rle with new method : 0.15860891342163086 time for calcul the mask position with numpy : 0.006574869155883789 nb_pixel_total : 122922 time to create 1 rle with old method : 0.12632346153259277 time for calcul the mask position with numpy : 0.006088972091674805 nb_pixel_total : 4059 time to create 1 rle with old method : 0.004232883453369141 time for calcul the mask position with numpy : 0.011626958847045898 nb_pixel_total : 725325 time to create 1 rle with new method : 0.13669800758361816 time for calcul the mask position with numpy : 0.007546901702880859 nb_pixel_total : 174412 time to create 1 rle with new method : 0.02760624885559082 create new chi : 0.528573751449585 time to delete rle : 0.0006213188171386719 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 6345 TO DO : save crop sub photo not yet done ! save time : 0.4224276542663574 map_output_result : {1339522520: (0.0, 'Should be the crop_list due to order', 0), 1339522480: (0.0, 'Should be the crop_list due to order', 0), 1339522479: (0.0, 'Should be the crop_list due to order', 0), 1339522440: (0.0, 'Should be the crop_list due to order', 0), 1339522439: (0.0, 'Should be the crop_list due to order', 0), 1339522332: (0.0, 'Should be the crop_list due to order', 0), 1339522331: (0.0, 'Should be the crop_list due to order', 0), 1339522295: (0.0, 'Should be the crop_list due to order', 0), 1339522294: (0.0, 'Should be the crop_list due to order', 0), 1339522262: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1339522520, 1339522480, 1339522479, 1339522440, 1339522439, 1339522332, 1339522331, 1339522295, 1339522294, 1339522262] Looping around the photos to save general results len do output : 10 /1339522520.Didn't retrieve data . /1339522480.Didn't retrieve data . /1339522479.Didn't retrieve data . /1339522440.Didn't retrieve data . /1339522439.Didn't retrieve data . /1339522332.Didn't retrieve data . /1339522331.Didn't retrieve data . /1339522295.Didn't retrieve data . /1339522294.Didn't retrieve data . /1339522262.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522520', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522480', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522479', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522440', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522439', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522332', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522331', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522295', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522294', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522262', None, None, None, None, None, '2612418') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013700246810913086 save_final save missing photos in datou_result : time spend for datou_step_exec : 16.25691294670105 time spend to save output : 0.014224052429199219 total time spend for step 5 : 16.27113699913025 step6:crop_condition Tue Feb 25 10:45:09 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 54 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740476710_341699 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6373324394226074 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740476711_341699 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.0514564514160156 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740476712_341699 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5645012855529785 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740476714_341699 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5360620021820068 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740476715_341699 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6786134243011475 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740476716_341699 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8319568634033203 we have finished the crop for the class : metal delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1339522520, 1339522480, 1339522479, 1339522440, 1339522439, 1339522332, 1339522331, 1339522295, 1339522294, 1339522262] Looping around the photos to save general results len do output : 7 /1339525882Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339525883Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339525884Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339525885Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339525886Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339525887Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339525888Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522520', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522480', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522479', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522440', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522439', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522332', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522331', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522295', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522294', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522262', None, None, None, None, None, '2612418') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 31 time used for this insertion : 0.02330780029296875 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.262271881103516 time spend to save output : 0.023858070373535156 total time spend for step 6 : 7.286129951477051 step7:ventilate_hashtags_in_portfolio Tue Feb 25 10:45:16 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20799180 get user id for portfolio 20799180 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20799180 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','mal_croppe','Teint_Dans_La_Masse','kraft','Carton_gris','flou','autre_refus','metal','plastique','cartonnette','papier','Carton_brun')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20799180 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','mal_croppe','Teint_Dans_La_Masse','kraft','Carton_gris','flou','autre_refus','metal','plastique','cartonnette','papier','Carton_brun')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20799333,20799334,20799335,20799336,20799337,20799338,20799339,20799340,20799341,20799342,20799343,20799344?tags=papier,metal,cartonnette,Teint_Dans_La_Masse,plastique,flou,environnement,Carton_gris,autre_refus,Carton_brun,kraft,mal_croppe Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1339522520, 1339522480, 1339522479, 1339522440, 1339522439, 1339522332, 1339522331, 1339522295, 1339522294, 1339522262] Looping around the photos to save general results len do output : 1 /20799180. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522520', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522480', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522479', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522440', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522439', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522332', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522331', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522295', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522294', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522262', None, None, None, None, None, '2612418') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.015044689178466797 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.4427614212036133 time spend to save output : 0.015273571014404297 total time spend for step 7 : 0.4580349922180176 step8:final Tue Feb 25 10:45:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1339522520: ('0.032359800219632175',), 1339522480: ('0.032359800219632175',), 1339522479: ('0.032359800219632175',), 1339522440: ('0.032359800219632175',), 1339522439: ('0.032359800219632175',), 1339522332: ('0.032359800219632175',), 1339522331: ('0.032359800219632175',), 1339522295: ('0.032359800219632175',), 1339522294: ('0.032359800219632175',), 1339522262: ('0.032359800219632175',)} new output for save of step final : {1339522520: ('0.032359800219632175',), 1339522480: ('0.032359800219632175',), 1339522479: ('0.032359800219632175',), 1339522440: ('0.032359800219632175',), 1339522439: ('0.032359800219632175',), 1339522332: ('0.032359800219632175',), 1339522331: ('0.032359800219632175',), 1339522295: ('0.032359800219632175',), 1339522294: ('0.032359800219632175',), 1339522262: ('0.032359800219632175',)} [1339522520, 1339522480, 1339522479, 1339522440, 1339522439, 1339522332, 1339522331, 1339522295, 1339522294, 1339522262] Looping around the photos to save general results len do output : 10 /1339522520.Didn't retrieve data . /1339522480.Didn't retrieve data . /1339522479.Didn't retrieve data . /1339522440.Didn't retrieve data . /1339522439.Didn't retrieve data . /1339522332.Didn't retrieve data . /1339522331.Didn't retrieve data . /1339522295.Didn't retrieve data . /1339522294.Didn't retrieve data . /1339522262.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522520', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522480', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522479', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522440', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522439', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522332', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522331', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522295', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522294', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522262', None, None, None, None, None, '2612418') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013448715209960938 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10122013092041016 time spend to save output : 0.013941764831542969 total time spend for step 8 : 0.11516189575195312 step9:velours_tree Tue Feb 25 10:45:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.03670144081115723 time spend to save output : 4.601478576660156e-05 total time spend for step 9 : 0.03674745559692383 step10:send_mail_cod Tue Feb 25 10:45:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20799180_25-02-2025_10_45_17.pdf 20799334 change filename to text .imagette207993341740476717 20799335 change filename to text .change filename to text .change filename to text .change filename to text .imagette207993351740476717 20799337 change filename to text .change filename to text .imagette207993371740476717 20799338 imagette207993381740476717 20799341 imagette207993411740476717 20799343 imagette207993431740476718 20799344 imagette207993441740476718 20799336 change filename to text .change filename to text .imagette207993361740476718 20799340 change filename to text .change filename to text .imagette207993401740476718 20799342 change filename to text .imagette207993421740476718 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20799180 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20799333,20799334,20799335,20799336,20799337,20799338,20799339,20799340,20799341,20799342,20799343,20799344?tags=papier,metal,cartonnette,Teint_Dans_La_Masse,plastique,flou,environnement,Carton_gris,autre_refus,Carton_brun,kraft,mal_croppe your option no_mail is active, we will not send the real mail to your client args[1339522520] : ('0.032359800219632175',) no score found for photo 1339522520 We are sending mail with results at report@fotonower.com args[1339522480] : ('0.032359800219632175',) no score found for photo 1339522480 We are sending mail with results at report@fotonower.com args[1339522479] : ('0.032359800219632175',) no score found for photo 1339522479 We are sending mail with results at report@fotonower.com args[1339522440] : ('0.032359800219632175',) no score found for photo 1339522440 We are sending mail with results at report@fotonower.com args[1339522439] : ('0.032359800219632175',) no score found for photo 1339522439 We are sending mail with results at report@fotonower.com args[1339522332] : ('0.032359800219632175',) no score found for photo 1339522332 We are sending mail with results at report@fotonower.com args[1339522331] : ('0.032359800219632175',) no score found for photo 1339522331 We are sending mail with results at report@fotonower.com args[1339522295] : ('0.032359800219632175',) no score found for photo 1339522295 We are sending mail with results at report@fotonower.com args[1339522294] : ('0.032359800219632175',) no score found for photo 1339522294 We are sending mail with results at report@fotonower.com args[1339522262] : ('0.032359800219632175',) no score found for photo 1339522262 We are sending mail with results at report@fotonower.com refus_total : 0.032359800219632175 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20799180 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1339522295,1339522261,1339522440,1339522698,1339522657,1339522656,1339522628,1339522627,1339522601,1339522600,1339522574,1339522572,1339522521,1339522520,1339522480,1339522479,1339521762,1339522439,1339522332,1339522331) Found this number of photos: 20 begin to download photo : 1339522295 begin to download photo : 1339522656 begin to download photo : 1339522574 begin to download photo : 1339522479 download finish for photo 1339522656 begin to download photo : 1339522628 download finish for photo 1339522295 begin to download photo : 1339522261 download finish for photo 1339522479 begin to download photo : 1339521762 download finish for photo 1339522574 begin to download photo : 1339522572 download finish for photo 1339522261 begin to download photo : 1339522440 download finish for photo 1339521762 begin to download photo : 1339522439 download finish for photo 1339522572 begin to download photo : 1339522521 download finish for photo 1339522628 begin to download photo : 1339522627 download finish for photo 1339522440 begin to download photo : 1339522698 download finish for photo 1339522521 begin to download photo : 1339522520 download finish for photo 1339522627 begin to download photo : 1339522601 download finish for photo 1339522439 begin to download photo : 1339522332 download finish for photo 1339522698 begin to download photo : 1339522657 download finish for photo 1339522520 begin to download photo : 1339522480 download finish for photo 1339522332 begin to download photo : 1339522331 download finish for photo 1339522601 begin to download photo : 1339522600 download finish for photo 1339522657 download finish for photo 1339522480 download finish for photo 1339522600 download finish for photo 1339522331 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20799180_25-02-2025_10_45_17.pdf results_Auto_P20799180_25-02-2025_10_45_17.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20799180_25-02-2025_10_45_17.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20799180','results_Auto_P20799180_25-02-2025_10_45_17.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20799180_25-02-2025_10_45_17.pdf','pdf','','0.23','0.032359800219632175') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1339522520, 1339522480, 1339522479, 1339522440, 1339522439, 1339522332, 1339522331, 1339522295, 1339522294, 1339522262] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522520', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522480', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522479', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522440', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522439', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522332', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522331', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522295', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522294', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522262', None, None, None, None, None, '2612418') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.016013145446777344 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.475492238998413 time spend to save output : 0.0162351131439209 total time spend for step 10 : 3.491727352142334 step11:split_time_score Tue Feb 25 10:45:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 28),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 25022025 20799180 Nombre de photos uploadées : 28 / 23040 (0%) 25022025 20799180 Nombre de photos taguées (types de déchets): 0 / 28 (0%) 25022025 20799180 Nombre de photos taguées (volume) : 0 / 28 (0%) elapsed_time : load_data_split_time_score 1.6689300537109375e-06 elapsed_time : order_list_meta_photo_and_scores 4.0531158447265625e-06 ???????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0012412071228027344 elapsed_time : insert_dashboard_record_day_entry 0.02369546890258789 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.027519706039487488 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20792421_25-02-2025_07_41_14.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20792421 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20792421 AND mptpi.`type`=3726 To do Qualite : 0.04279993537833964 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20793621_25-02-2025_08_15_36.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20793621 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20793621 AND mptpi.`type`=3726 To do Qualite : 0.04836044824494673 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20793622_25-02-2025_08_22_36.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20793622 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20793622 AND mptpi.`type`=3726 To do Qualite : 0.023273948616390653 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20793623_25-02-2025_08_11_27.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20793623 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20793623 AND mptpi.`type`=3726 To do Qualite : 0.04504098060841874 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20795491_25-02-2025_09_51_32.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20795491 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20795491 AND mptpi.`type`=3726 To do Qualite : 0.041251844000054855 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20795492_25-02-2025_10_02_08.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20795492 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20795492 AND mptpi.`type`=3726 To do Qualite : 0.04903814151177415 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20796642_25-02-2025_10_33_36.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20796642 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20796642 AND mptpi.`type`=3726 To do Qualite : 0.054504090865715456 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20796643_25-02-2025_09_21_17.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20796643 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20796643 AND mptpi.`type`=3726 To do Qualite : 0.03200016630010863 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20797866_25-02-2025_10_11_47.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20797866 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20797866 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20799179 order by id desc limit 1 Qualite : 0.036218299384119884 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20799180_25-02-2025_10_45_17.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20799180 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20799180 AND mptpi.`type`=3726 To do Qualite : 0.03480926688887944 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20799181_25-02-2025_10_42_17.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20799181 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20799181 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'25022025': {'nb_upload': 28, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1339522520, 1339522480, 1339522479, 1339522440, 1339522439, 1339522332, 1339522331, 1339522295, 1339522294, 1339522262] Looping around the photos to save general results len do output : 1 /20799180Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522520', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522480', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522479', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522440', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522439', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522332', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522331', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522295', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522294', None, None, None, None, None, '2612418') ('3459', None, None, None, None, None, None, None, '2612418') ('3459', None, '1339522262', None, None, None, None, None, '2612418') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01718306541442871 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.641829252243042 time spend to save output : 0.017426490783691406 total time spend for step 11 : 1.6592557430267334 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 36.47user 28.12system 2:57.17elapsed 36%CPU (0avgtext+0avgdata 2595832maxresident)k 236648inputs+27520outputs (883major+1930314minor)pagefaults 0swaps