python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 768900 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 29>10 please execute split_portfolio.py -i 20586312 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2590787'] with mtr_portfolio_ids : ['20586312'] and first list_photo_ids : [] new path : /proc/768900/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.1417930126190186 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Sat Feb 15 12:06:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 2283 wait 20 seconds l 3637 free memory gpu now : 2283 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-15 12:06:51.539499: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-15 12:06:51.567245: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-15 12:06:51.569487: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fc838000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-15 12:06:51.569551: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-15 12:06:51.573893: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-15 12:06:51.843809: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x14498780 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-15 12:06:51.843882: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-15 12:06:51.848395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-15 12:06:51.849846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-15 12:06:51.854808: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-15 12:06:51.858016: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-15 12:06:51.859459: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-15 12:06:51.863658: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-15 12:06:51.865237: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-15 12:06:51.872183: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-15 12:06:51.873332: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-15 12:06:51.873423: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-15 12:06:51.874011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-15 12:06:51.874027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-15 12:06:51.874037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-15 12:06:51.875039: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3024 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-15 12:06:52.343914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-15 12:06:52.344005: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-15 12:06:52.344023: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-15 12:06:52.344038: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-15 12:06:52.344053: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-15 12:06:52.344068: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-15 12:06:52.344083: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-15 12:06:52.344098: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-15 12:06:52.344957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-15 12:06:52.345869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-15 12:06:52.345910: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-15 12:06:52.345928: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-15 12:06:52.345944: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-15 12:06:52.345960: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-15 12:06:52.345976: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-15 12:06:52.345993: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-15 12:06:52.346009: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-15 12:06:52.346890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-15 12:06:52.346938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-15 12:06:52.346947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-15 12:06:52.346955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-15 12:06:52.347855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3024 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-15 12:07:00.520804: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-15 12:07:00.707735: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-15 12:07:02.142680: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.143374: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.21G (2372390144 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.143407: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-15 12:07:02.144036: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.144052: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-15 12:07:02.151167: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.151189: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-15 12:07:02.151814: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.151830: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-15 12:07:02.158362: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.158383: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-15 12:07:02.158972: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.158992: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-15 12:07:02.189924: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.189956: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-15 12:07:02.190552: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.190567: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-15 12:07:02.196458: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.196480: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-15 12:07:02.197068: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.197083: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-15 12:07:02.230745: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.231354: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.233232: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.233822: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.278667: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.279371: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.281682: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.282315: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.312208: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.312856: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.314497: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.315155: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.320869: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.321505: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.323263: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.323897: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.329661: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.330297: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.331924: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.332558: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.359326: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.359978: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.360606: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.361232: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.364867: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.365500: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.381025: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.381672: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.382299: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.382948: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.395357: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.395997: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.396623: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.397260: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.401633: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.402267: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.407067: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.407742: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.420109: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.420757: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.424990: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.425732: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.426363: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.427017: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.427849: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.428483: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.439110: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.439764: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.440414: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.441042: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.441667: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.442294: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.457694: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.458328: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.514437: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.515091: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.515695: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.516281: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.524102: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.524690: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.571554: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.572196: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.572788: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.573365: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.577416: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.604993: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.606133: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.607302: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.609350: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.609374: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-02-15 12:07:02.618891: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.619772: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.628642: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.629213: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.629794: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.630363: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.630960: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-15 12:07:02.631590: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2635988992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 14.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 14.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 5.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 4.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 Detection mask done ! Trying to reset tf kernel 769869 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2283 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3476 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.012519121170043945 nb_pixel_total : 584434 time to create 1 rle with new method : 0.05272197723388672 length of segment : 1238 time for calcul the mask position with numpy : 0.00029778480529785156 nb_pixel_total : 13948 time to create 1 rle with old method : 0.0167391300201416 length of segment : 303 time for calcul the mask position with numpy : 0.0029947757720947266 nb_pixel_total : 170484 time to create 1 rle with new method : 0.009002685546875 length of segment : 963 time for calcul the mask position with numpy : 0.0009176731109619141 nb_pixel_total : 62809 time to create 1 rle with old method : 0.07156634330749512 length of segment : 296 time for calcul the mask position with numpy : 0.0012097358703613281 nb_pixel_total : 98610 time to create 1 rle with old method : 0.10860586166381836 length of segment : 469 time for calcul the mask position with numpy : 0.0008130073547363281 nb_pixel_total : 64031 time to create 1 rle with old method : 0.07278084754943848 length of segment : 199 time for calcul the mask position with numpy : 0.00641632080078125 nb_pixel_total : 192802 time to create 1 rle with new method : 0.030139923095703125 length of segment : 1108 time for calcul the mask position with numpy : 0.003316164016723633 nb_pixel_total : 285423 time to create 1 rle with new method : 0.007110118865966797 length of segment : 856 time for calcul the mask position with numpy : 0.004540443420410156 nb_pixel_total : 349869 time to create 1 rle with new method : 0.016277551651000977 length of segment : 1002 time for calcul the mask position with numpy : 0.0009784698486328125 nb_pixel_total : 63234 time to create 1 rle with old method : 0.07109618186950684 length of segment : 513 time for calcul the mask position with numpy : 0.0003597736358642578 nb_pixel_total : 20169 time to create 1 rle with old method : 0.024109363555908203 length of segment : 129 time for calcul the mask position with numpy : 0.0004482269287109375 nb_pixel_total : 32477 time to create 1 rle with old method : 0.03835296630859375 length of segment : 171 time for calcul the mask position with numpy : 0.005331754684448242 nb_pixel_total : 271270 time to create 1 rle with new method : 0.01111459732055664 length of segment : 783 time for calcul the mask position with numpy : 0.0010385513305664062 nb_pixel_total : 62983 time to create 1 rle with old method : 0.06893444061279297 length of segment : 332 time for calcul the mask position with numpy : 0.015322446823120117 nb_pixel_total : 978222 time to create 1 rle with new method : 0.158522367477417 length of segment : 1811 time for calcul the mask position with numpy : 0.002050638198852539 nb_pixel_total : 165995 time to create 1 rle with new method : 0.004351377487182617 length of segment : 359 time for calcul the mask position with numpy : 0.00020122528076171875 nb_pixel_total : 9883 time to create 1 rle with old method : 0.011342048645019531 length of segment : 264 time for calcul the mask position with numpy : 0.0021085739135742188 nb_pixel_total : 115408 time to create 1 rle with old method : 0.1263880729675293 length of segment : 599 time for calcul the mask position with numpy : 0.00031757354736328125 nb_pixel_total : 6501 time to create 1 rle with old method : 0.007345676422119141 length of segment : 141 time for calcul the mask position with numpy : 0.0010001659393310547 nb_pixel_total : 31848 time to create 1 rle with old method : 0.03601789474487305 length of segment : 316 time for calcul the mask position with numpy : 0.00011873245239257812 nb_pixel_total : 766 time to create 1 rle with old method : 0.0010828971862792969 length of segment : 51 time for calcul the mask position with numpy : 0.005033016204833984 nb_pixel_total : 321343 time to create 1 rle with new method : 0.012066841125488281 length of segment : 1263 time for calcul the mask position with numpy : 0.00022220611572265625 nb_pixel_total : 2565 time to create 1 rle with old method : 0.0031070709228515625 length of segment : 99 time for calcul the mask position with numpy : 0.0021593570709228516 nb_pixel_total : 88400 time to create 1 rle with old method : 0.09526681900024414 length of segment : 653 time for calcul the mask position with numpy : 0.006334543228149414 nb_pixel_total : 261695 time to create 1 rle with new method : 0.03983283042907715 length of segment : 985 time for calcul the mask position with numpy : 0.0003178119659423828 nb_pixel_total : 10537 time to create 1 rle with old method : 0.012257575988769531 length of segment : 93 time for calcul the mask position with numpy : 0.0034873485565185547 nb_pixel_total : 284341 time to create 1 rle with new method : 0.008603334426879883 length of segment : 1319 time for calcul the mask position with numpy : 0.0008749961853027344 nb_pixel_total : 41132 time to create 1 rle with old method : 0.04683184623718262 length of segment : 205 time for calcul the mask position with numpy : 0.014270544052124023 nb_pixel_total : 866729 time to create 1 rle with new method : 0.07744550704956055 length of segment : 1526 time for calcul the mask position with numpy : 0.0013287067413330078 nb_pixel_total : 101761 time to create 1 rle with old method : 0.12544536590576172 length of segment : 451 time for calcul the mask position with numpy : 0.0013778209686279297 nb_pixel_total : 98428 time to create 1 rle with old method : 0.10597825050354004 length of segment : 443 time for calcul the mask position with numpy : 0.0004322528839111328 nb_pixel_total : 14985 time to create 1 rle with old method : 0.018310070037841797 length of segment : 72 time for calcul the mask position with numpy : 0.00019025802612304688 nb_pixel_total : 5190 time to create 1 rle with old method : 0.00620722770690918 length of segment : 76 time for calcul the mask position with numpy : 0.010656595230102539 nb_pixel_total : 459012 time to create 1 rle with new method : 0.0395510196685791 length of segment : 1063 time for calcul the mask position with numpy : 0.00010228157043457031 nb_pixel_total : 1327 time to create 1 rle with old method : 0.0016465187072753906 length of segment : 97 time for calcul the mask position with numpy : 0.00016808509826660156 nb_pixel_total : 7224 time to create 1 rle with old method : 0.008641481399536133 length of segment : 111 time for calcul the mask position with numpy : 0.0005559921264648438 nb_pixel_total : 13941 time to create 1 rle with old method : 0.015914440155029297 length of segment : 285 time for calcul the mask position with numpy : 0.017123937606811523 nb_pixel_total : 873987 time to create 1 rle with new method : 0.051229000091552734 length of segment : 1052 time for calcul the mask position with numpy : 0.0014824867248535156 nb_pixel_total : 107268 time to create 1 rle with old method : 0.11782002449035645 length of segment : 758 time for calcul the mask position with numpy : 0.002899169921875 nb_pixel_total : 180640 time to create 1 rle with new method : 0.0055065155029296875 length of segment : 652 time for calcul the mask position with numpy : 0.0666511058807373 nb_pixel_total : 929913 time to create 1 rle with new method : 0.0737147331237793 length of segment : 1697 time for calcul the mask position with numpy : 0.007285356521606445 nb_pixel_total : 289230 time to create 1 rle with new method : 0.015441417694091797 length of segment : 1003 time for calcul the mask position with numpy : 0.002318859100341797 nb_pixel_total : 171268 time to create 1 rle with new method : 0.0050318241119384766 length of segment : 541 time for calcul the mask position with numpy : 0.00027871131896972656 nb_pixel_total : 4944 time to create 1 rle with old method : 0.006254911422729492 length of segment : 58 time for calcul the mask position with numpy : 0.007269620895385742 nb_pixel_total : 491812 time to create 1 rle with new method : 0.04143834114074707 length of segment : 1285 time spent for convertir_results : 6.644945383071899 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 57 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 27690 save missing photos in datou_result : time spend for datou_step_exec : 50.33170700073242 time spend to save output : 1.5411419868469238 total time spend for step 1 : 51.872848987579346 step2:crop_condition Sat Feb 15 12:07:19 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 57 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1337867925, 1337867903, 1337867902, 1337867885, 1337867869, 1337867868, 1337867821, 1337867820, 1337867590, 1337867589] Looping around the photos to save general results len do output : 19 /-3676540605Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540611Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540615Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540612Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676569776Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540627Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540628Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540634Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540635Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540595Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540626Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676544244Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540601Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540604Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540598Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540614Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676569770Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540607Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3676540625Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867925', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867903', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867902', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867885', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867869', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867868', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867821', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867820', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867590', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867589', None, None, None, None, None, '2590787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 67 time used for this insertion : 0.02209305763244629 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.9727325439453125 time spend to save output : 0.023318767547607422 total time spend for step 2 : 7.99605131149292 step3:thcl Sat Feb 15 12:07:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.011394023895263672 time to convert the images to numpy array : 0.014623403549194336 time to import caffe and check if the image exist : 0.010953664779663086 time to convert the images to numpy array : 0.025896072387695312 time to import caffe and check if the image exist : 0.014268159866333008 time to convert the images to numpy array : 0.023375272750854492 time to import caffe and check if the image exist : 0.01139068603515625 time to convert the images to numpy array : 0.030398130416870117 time to import caffe and check if the image exist : 0.01167607307434082 time to convert the images to numpy array : 0.031760454177856445 time to import caffe and check if the image exist : 0.011232614517211914 time to convert the images to numpy array : 0.03271794319152832 time to import caffe and check if the image exist : 0.015065908432006836 time to convert the images to numpy array : 0.0416417121887207 time to import caffe and check if the image exist : 0.010781526565551758 time to convert the images to numpy array : 0.0518498420715332 time to import caffe and check if the image exist : 0.012388229370117188 time to convert the images to numpy array : 0.05810260772705078 time to import caffe and check if the image exist : 0.011976003646850586 time to convert the images to numpy array : 0.07287216186523438 total time to convert the images to numpy array : 0.5302543640136719 list photo_ids error: [] list photo_ids correct : [-3676540615, -3676540612, -3676540625, -3676540598, -3676540614, -3676540626, -3676544244, -3676540628, -3676540634, -3676569770, -3676540607, -3676540635, -3676540595, -3676540605, -3676540611, -3676540601, -3676540604, -3676569776, -3676540627] number of photos to traite : 19 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3476 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3257 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.22792649269104004 time used to do the prediction : 0.12918329238891602 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3255 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3255 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.2204883098602295 time used to do the prediction : 0.05848217010498047 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 5.9604644775390625e-06 save missing photos in datou_result : time spend for datou_step_exec : 8.258933067321777 time spend to save output : 0.00023627281188964844 total time spend for step 3 : 8.259169340133667 step4:merge_mask_thcl_custom Sat Feb 15 12:07:35 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 57 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1337867925, 1337867903, 1337867902, 1337867885, 1337867869, 1337867868, 1337867821, 1337867820, 1337867590, 1337867589] Looping around the photos to save general results len do output : 10 /1337867925Didn't retrieve data .Didn't retrieve data . /1337867903Didn't retrieve data .Didn't retrieve data . /1337867902Didn't retrieve data .Didn't retrieve data . /1337867885Didn't retrieve data .Didn't retrieve data . /1337867869Didn't retrieve data .Didn't retrieve data . /1337867868Didn't retrieve data .Didn't retrieve data . /1337867821Didn't retrieve data .Didn't retrieve data . /1337867820Didn't retrieve data .Didn't retrieve data . /1337867590Didn't retrieve data .Didn't retrieve data . /1337867589Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867925', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867903', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867902', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867885', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867869', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867868', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867821', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867820', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867590', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867589', None, None, None, None, None, '2590787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.023447513580322266 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.051824092864990234 time spend to save output : 0.024012088775634766 total time spend for step 4 : 0.075836181640625 step5:rle_unique_nms_with_priority Sat Feb 15 12:07:35 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 0.9320576190948486 time for calcul the mask position with numpy : 0.04596114158630371 nb_pixel_total : 1460241 time to create 1 rle with new method : 0.35265302658081055 time for calcul the mask position with numpy : 0.006598949432373047 nb_pixel_total : 2520 time to create 1 rle with old method : 0.003621816635131836 time for calcul the mask position with numpy : 0.011865377426147461 nb_pixel_total : 596890 time to create 1 rle with new method : 0.19454240798950195 time for calcul the mask position with numpy : 0.006435871124267578 nb_pixel_total : 13949 time to create 1 rle with old method : 0.015366077423095703 create new chi : 0.6540887355804443 time to delete rle : 0.018209457397460938 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 3803 TO DO : save crop sub photo not yet done ! save time : 0.34732890129089355 nb_obj : 10 nb_hashtags : 5 time to prepare the origin masks : 2.1356170177459717 time for calcul the mask position with numpy : 0.018917560577392578 nb_pixel_total : 1300075 time to create 1 rle with new method : 0.30182862281799316 time for calcul the mask position with numpy : 0.008597612380981445 nb_pixel_total : 280751 time to create 1 rle with new method : 0.08354306221008301 time for calcul the mask position with numpy : 0.007386445999145508 nb_pixel_total : 44435 time to create 1 rle with old method : 0.048700809478759766 time for calcul the mask position with numpy : 0.006708383560180664 nb_pixel_total : 37648 time to create 1 rle with old method : 0.042696237564086914 time for calcul the mask position with numpy : 0.006566047668457031 nb_pixel_total : 20067 time to create 1 rle with old method : 0.027872323989868164 time for calcul the mask position with numpy : 0.006987333297729492 nb_pixel_total : 24333 time to create 1 rle with old method : 0.02774643898010254 time for calcul the mask position with numpy : 0.006819725036621094 nb_pixel_total : 67817 time to create 1 rle with old method : 0.07720041275024414 time for calcul the mask position with numpy : 0.006894111633300781 nb_pixel_total : 62598 time to create 1 rle with old method : 0.06860470771789551 time for calcul the mask position with numpy : 0.0069119930267333984 nb_pixel_total : 1309 time to create 1 rle with old method : 0.002187967300415039 time for calcul the mask position with numpy : 0.007719755172729492 nb_pixel_total : 170614 time to create 1 rle with new method : 0.1477797031402588 time for calcul the mask position with numpy : 0.0075261592864990234 nb_pixel_total : 63953 time to create 1 rle with old method : 0.06940650939941406 create new chi : 1.0093846321105957 time to delete rle : 0.0008416175842285156 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 9367 TO DO : save crop sub photo not yet done ! save time : 0.7020659446716309 nb_obj : 4 nb_hashtags : 3 time to prepare the origin masks : 0.48873448371887207 time for calcul the mask position with numpy : 0.01362299919128418 nb_pixel_total : 1008695 time to create 1 rle with new method : 0.032502174377441406 time for calcul the mask position with numpy : 0.0061817169189453125 nb_pixel_total : 13358 time to create 1 rle with old method : 0.015308141708374023 time for calcul the mask position with numpy : 0.014575004577636719 nb_pixel_total : 747998 time to create 1 rle with new method : 0.029720783233642578 time for calcul the mask position with numpy : 0.007510662078857422 nb_pixel_total : 271115 time to create 1 rle with new method : 0.029067277908325195 time for calcul the mask position with numpy : 0.006346940994262695 nb_pixel_total : 32434 time to create 1 rle with old method : 0.03852653503417969 create new chi : 0.19434165954589844 time to delete rle : 0.0005857944488525391 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 5901 TO DO : save crop sub photo not yet done ! save time : 0.4832162857055664 nb_obj : 9 nb_hashtags : 3 time to prepare the origin masks : 0.7553596496582031 time for calcul the mask position with numpy : 0.3091702461242676 nb_pixel_total : 1739590 time to create 1 rle with new method : 0.23196196556091309 time for calcul the mask position with numpy : 0.0070037841796875 nb_pixel_total : 13 time to create 1 rle with old method : 8.535385131835938e-05 time for calcul the mask position with numpy : 0.011680364608764648 nb_pixel_total : 2575 time to create 1 rle with old method : 0.002959728240966797 time for calcul the mask position with numpy : 0.01165628433227539 nb_pixel_total : 9810 time to create 1 rle with old method : 0.011591672897338867 time for calcul the mask position with numpy : 0.010214090347290039 nb_pixel_total : 765 time to create 1 rle with old method : 0.0008864402770996094 time for calcul the mask position with numpy : 0.010337591171264648 nb_pixel_total : 6468 time to create 1 rle with old method : 0.007200717926025391 time for calcul the mask position with numpy : 0.010346412658691406 nb_pixel_total : 3924 time to create 1 rle with old method : 0.004682064056396484 time for calcul the mask position with numpy : 0.01103067398071289 nb_pixel_total : 112581 time to create 1 rle with old method : 0.12380576133728027 time for calcul the mask position with numpy : 0.012224435806274414 nb_pixel_total : 166062 time to create 1 rle with new method : 0.04440760612487793 time for calcul the mask position with numpy : 0.010634422302246094 nb_pixel_total : 31812 time to create 1 rle with old method : 0.03516221046447754 create new chi : 0.8708078861236572 time to delete rle : 0.0006265640258789062 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 5239 TO DO : save crop sub photo not yet done ! save time : 0.442629337310791 nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.43416762351989746 time for calcul the mask position with numpy : 0.017923355102539062 nb_pixel_total : 1561606 time to create 1 rle with new method : 0.031442880630493164 time for calcul the mask position with numpy : 0.006686210632324219 nb_pixel_total : 10521 time to create 1 rle with old method : 0.011521339416503906 time for calcul the mask position with numpy : 0.0072002410888671875 nb_pixel_total : 129447 time to create 1 rle with old method : 0.13980555534362793 time for calcul the mask position with numpy : 0.007974863052368164 nb_pixel_total : 88293 time to create 1 rle with old method : 0.09530019760131836 time for calcul the mask position with numpy : 0.009063959121704102 nb_pixel_total : 283733 time to create 1 rle with new method : 0.0347135066986084 create new chi : 0.37268543243408203 time to delete rle : 0.0006084442138671875 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6329 TO DO : save crop sub photo not yet done ! save time : 0.547274112701416 nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.3591337203979492 time for calcul the mask position with numpy : 0.014693737030029297 nb_pixel_total : 1086428 time to create 1 rle with new method : 0.2323322296142578 time for calcul the mask position with numpy : 0.013132333755493164 nb_pixel_total : 842811 time to create 1 rle with new method : 0.2153151035308838 time for calcul the mask position with numpy : 0.006770133972167969 nb_pixel_total : 40936 time to create 1 rle with old method : 0.04377007484436035 time for calcul the mask position with numpy : 0.006749391555786133 nb_pixel_total : 5256 time to create 1 rle with old method : 0.006307125091552734 time for calcul the mask position with numpy : 0.006929159164428711 nb_pixel_total : 98169 time to create 1 rle with old method : 0.10857367515563965 create new chi : 0.6622087955474854 time to delete rle : 0.0010251998901367188 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 4936 TO DO : save crop sub photo not yet done ! save time : 0.41884732246398926 nb_obj : 5 nb_hashtags : 2 time to prepare the origin masks : 0.42608213424682617 time for calcul the mask position with numpy : 0.01706552505493164 nb_pixel_total : 1591652 time to create 1 rle with new method : 0.07799768447875977 time for calcul the mask position with numpy : 0.006189107894897461 nb_pixel_total : 1320 time to create 1 rle with old method : 0.0014128684997558594 time for calcul the mask position with numpy : 0.006170034408569336 nb_pixel_total : 1668 time to create 1 rle with old method : 0.0025479793548583984 time for calcul the mask position with numpy : 0.009114503860473633 nb_pixel_total : 458958 time to create 1 rle with new method : 0.29313135147094727 time for calcul the mask position with numpy : 0.011397361755371094 nb_pixel_total : 5179 time to create 1 rle with old method : 0.00822758674621582 time for calcul the mask position with numpy : 0.01162266731262207 nb_pixel_total : 14823 time to create 1 rle with old method : 0.02347278594970703 create new chi : 0.4784865379333496 time to delete rle : 0.0006589889526367188 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 4635 TO DO : save crop sub photo not yet done ! save time : 0.41811084747314453 nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 0.37421417236328125 time for calcul the mask position with numpy : 0.015443801879882812 nb_pixel_total : 1080524 time to create 1 rle with new method : 0.09313392639160156 time for calcul the mask position with numpy : 0.014725685119628906 nb_pixel_total : 873454 time to create 1 rle with new method : 0.12560415267944336 time for calcul the mask position with numpy : 0.006383180618286133 nb_pixel_total : 13925 time to create 1 rle with old method : 0.015365123748779297 time for calcul the mask position with numpy : 0.007166862487792969 nb_pixel_total : 105697 time to create 1 rle with old method : 0.11888313293457031 create new chi : 0.41587162017822266 time to delete rle : 0.000652313232421875 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 5202 TO DO : save crop sub photo not yet done ! save time : 0.41335344314575195 nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 0.5194900035858154 time for calcul the mask position with numpy : 0.032880306243896484 nb_pixel_total : 914110 time to create 1 rle with new method : 0.22057294845581055 time for calcul the mask position with numpy : 0.007435321807861328 nb_pixel_total : 177654 time to create 1 rle with new method : 0.0871431827545166 time for calcul the mask position with numpy : 0.006579160690307617 nb_pixel_total : 31076 time to create 1 rle with old method : 0.03443455696105957 time for calcul the mask position with numpy : 0.012431144714355469 nb_pixel_total : 950760 time to create 1 rle with new method : 0.1882944107055664 create new chi : 0.606036901473999 time to delete rle : 0.0006368160247802734 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6330 TO DO : save crop sub photo not yet done ! save time : 0.5406038761138916 nb_obj : 5 nb_hashtags : 2 time to prepare the origin masks : 0.5382547378540039 time for calcul the mask position with numpy : 0.058432817459106445 nb_pixel_total : 1264057 time to create 1 rle with new method : 0.31099987030029297 time for calcul the mask position with numpy : 0.007569789886474609 nb_pixel_total : 12 time to create 1 rle with old method : 6.0558319091796875e-05 time for calcul the mask position with numpy : 0.009642362594604492 nb_pixel_total : 347736 time to create 1 rle with new method : 0.11050987243652344 time for calcul the mask position with numpy : 0.007598876953125 nb_pixel_total : 171691 time to create 1 rle with new method : 0.03574109077453613 time for calcul the mask position with numpy : 0.006564140319824219 nb_pixel_total : 1153 time to create 1 rle with old method : 0.0017838478088378906 time for calcul the mask position with numpy : 0.010301351547241211 nb_pixel_total : 288951 time to create 1 rle with new method : 0.0363161563873291 create new chi : 0.6060981750488281 time to delete rle : 0.0008378028869628906 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 1518 TO DO : save crop sub photo not yet done ! save time : 0.2140648365020752 map_output_result : {1337867925: (0.0, 'Should be the crop_list due to order', 0.0), 1337867903: (0.0, 'Should be the crop_list due to order', 0.0), 1337867902: (0.0, 'Should be the crop_list due to order', 0.0), 1337867885: (0.0, 'Should be the crop_list due to order', 0.0), 1337867869: (0.0, 'Should be the crop_list due to order', 0.0), 1337867868: (0.0, 'Should be the crop_list due to order', 0.0), 1337867821: (0.0, 'Should be the crop_list due to order', 0.0), 1337867820: (0.0, 'Should be the crop_list due to order', 0.0), 1337867590: (0.0, 'Should be the crop_list due to order', 0.0), 1337867589: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1337867925, 1337867903, 1337867902, 1337867885, 1337867869, 1337867868, 1337867821, 1337867820, 1337867590, 1337867589] Looping around the photos to save general results len do output : 10 /1337867925.Didn't retrieve data . /1337867903.Didn't retrieve data . /1337867902.Didn't retrieve data . /1337867885.Didn't retrieve data . /1337867869.Didn't retrieve data . /1337867868.Didn't retrieve data . /1337867821.Didn't retrieve data . /1337867820.Didn't retrieve data . /1337867590.Didn't retrieve data . /1337867589.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867925', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867903', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867902', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867885', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867869', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867868', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867821', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867820', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867590', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867589', None, None, None, None, None, '2590787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01695728302001953 save_final save missing photos in datou_result : time spend for datou_step_exec : 17.692780256271362 time spend to save output : 0.017505407333374023 total time spend for step 5 : 17.710285663604736 step6:crop_condition Sat Feb 15 12:07:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 67 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739617676_768900 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.8622057437896729 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 map_result returned by crop_photo_return_map_crop : length : 0 About to insert : list_path_to_insert length 0 new photo from crops ! About to upload 0 photos WARNING : list_path_to_insert is empty, cannot upload ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739617678_768900 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6239941120147705 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739617679_768900 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5674009323120117 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1337867925, 1337867903, 1337867902, 1337867885, 1337867869, 1337867868, 1337867821, 1337867820, 1337867590, 1337867589] Looping around the photos to save general results len do output : 8 /1337876640Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337876641Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337876642Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337876643Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337876644Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337876645Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337876648Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337876649Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867925', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867903', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867902', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867885', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867869', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867868', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867821', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867820', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867590', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867589', None, None, None, None, None, '2590787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 34 time used for this insertion : 0.02218484878540039 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.877991676330566 time spend to save output : 0.02278876304626465 total time spend for step 6 : 6.900780439376831 step7:ventilate_hashtags_in_portfolio Sat Feb 15 12:08:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20586312 get user id for portfolio 20586312 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20586312 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_brun','mal_croppe','flou','autre_refus','plastique','papier','kraft','cartonnette','Carton_gris','metal','Teint_Dans_La_Masse','environnement')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20586312 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_brun','mal_croppe','flou','autre_refus','plastique','papier','kraft','cartonnette','Carton_gris','metal','Teint_Dans_La_Masse','environnement')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20586312 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_brun','mal_croppe','flou','autre_refus','plastique','papier','kraft','cartonnette','Carton_gris','metal','Teint_Dans_La_Masse','environnement')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20586315,20586316,20586317,20586318,20586319,20586320,20586321,20586322,20586323,20586324,20586325,20586326?tags=Carton_brun,mal_croppe,flou,autre_refus,plastique,papier,kraft,cartonnette,Carton_gris,metal,Teint_Dans_La_Masse,environnement Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1337867925, 1337867903, 1337867902, 1337867885, 1337867869, 1337867868, 1337867821, 1337867820, 1337867590, 1337867589] Looping around the photos to save general results len do output : 1 /20586312. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867925', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867903', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867902', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867885', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867869', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867868', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867821', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867820', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867590', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867589', None, None, None, None, None, '2590787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013200044631958008 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.698655366897583 time spend to save output : 0.013435602188110352 total time spend for step 7 : 1.7120909690856934 step8:final Sat Feb 15 12:08:02 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1337867925: ('0.0423778629895155',), 1337867903: ('0.0423778629895155',), 1337867902: ('0.0423778629895155',), 1337867885: ('0.0423778629895155',), 1337867869: ('0.0423778629895155',), 1337867868: ('0.0423778629895155',), 1337867821: ('0.0423778629895155',), 1337867820: ('0.0423778629895155',), 1337867590: ('0.0423778629895155',), 1337867589: ('0.0423778629895155',)} new output for save of step final : {1337867925: ('0.0423778629895155',), 1337867903: ('0.0423778629895155',), 1337867902: ('0.0423778629895155',), 1337867885: ('0.0423778629895155',), 1337867869: ('0.0423778629895155',), 1337867868: ('0.0423778629895155',), 1337867821: ('0.0423778629895155',), 1337867820: ('0.0423778629895155',), 1337867590: ('0.0423778629895155',), 1337867589: ('0.0423778629895155',)} [1337867925, 1337867903, 1337867902, 1337867885, 1337867869, 1337867868, 1337867821, 1337867820, 1337867590, 1337867589] Looping around the photos to save general results len do output : 10 /1337867925.Didn't retrieve data . /1337867903.Didn't retrieve data . /1337867902.Didn't retrieve data . /1337867885.Didn't retrieve data . /1337867869.Didn't retrieve data . /1337867868.Didn't retrieve data . /1337867821.Didn't retrieve data . /1337867820.Didn't retrieve data . /1337867590.Didn't retrieve data . /1337867589.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867925', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867903', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867902', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867885', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867869', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867868', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867821', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867820', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867590', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867589', None, None, None, None, None, '2590787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013648271560668945 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10511517524719238 time spend to save output : 0.014130115509033203 total time spend for step 8 : 0.11924529075622559 step9:velours_tree Sat Feb 15 12:08:02 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.10478329658508301 time spend to save output : 4.5299530029296875e-05 total time spend for step 9 : 0.1048285961151123 step10:send_mail_cod Sat Feb 15 12:08:02 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20586312_15-02-2025_12_08_02.pdf 20586316 imagette205863161739617682 20586317 imagette205863171739617682 20586318 change filename to text .imagette205863181739617682 20586319 change filename to text .change filename to text .imagette205863191739617682 20586321 imagette205863211739617682 20586322 change filename to text .change filename to text .imagette205863221739617682 20586324 imagette205863241739617682 20586315 change filename to text .imagette205863151739617682 20586323 imagette205863231739617683 20586325 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette205863251739617683 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20586312 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20586315,20586316,20586317,20586318,20586319,20586320,20586321,20586322,20586323,20586324,20586325,20586326?tags=Carton_brun,mal_croppe,flou,autre_refus,plastique,papier,kraft,cartonnette,Carton_gris,metal,Teint_Dans_La_Masse,environnement your option no_mail is active, we will not send the real mail to your client args[1337867925] : ('0.0423778629895155',) no score found for photo 1337867925 apple ('0.0423778629895155',) We are sending mail with results at report@fotonower.com args[1337867903] : ('0.0423778629895155',) no score found for photo 1337867903 apple ('0.0423778629895155',) We are sending mail with results at report@fotonower.com args[1337867902] : ('0.0423778629895155',) no score found for photo 1337867902 apple ('0.0423778629895155',) We are sending mail with results at report@fotonower.com args[1337867885] : ('0.0423778629895155',) no score found for photo 1337867885 apple ('0.0423778629895155',) We are sending mail with results at report@fotonower.com args[1337867869] : ('0.0423778629895155',) no score found for photo 1337867869 apple ('0.0423778629895155',) We are sending mail with results at report@fotonower.com args[1337867868] : ('0.0423778629895155',) no score found for photo 1337867868 apple ('0.0423778629895155',) We are sending mail with results at report@fotonower.com args[1337867821] : ('0.0423778629895155',) no score found for photo 1337867821 apple ('0.0423778629895155',) We are sending mail with results at report@fotonower.com args[1337867820] : ('0.0423778629895155',) no score found for photo 1337867820 apple ('0.0423778629895155',) We are sending mail with results at report@fotonower.com args[1337867590] : ('0.0423778629895155',) no score found for photo 1337867590 apple ('0.0423778629895155',) We are sending mail with results at report@fotonower.com args[1337867589] : ('0.0423778629895155',) no score found for photo 1337867589 apple ('0.0423778629895155',) We are sending mail with results at report@fotonower.com refus_total : 0.0423778629895155 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20586312 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1337867903,1337867590,1337867498,1337867925,1337867902,1337867885,1337867869,1337867868,1337867821,1337867820,1337867589,1337867566,1337867564,1337867545,1337867544,1337865541,1337867495,1337867218,1337867173,1337866950) Found this number of photos: 20 begin to download photo : 1337867903 begin to download photo : 1337867885 begin to download photo : 1337867589 begin to download photo : 1337865541 download finish for photo 1337867885 begin to download photo : 1337867869 download finish for photo 1337865541 begin to download photo : 1337867495 download finish for photo 1337867589 begin to download photo : 1337867566 download finish for photo 1337867903 begin to download photo : 1337867590 download finish for photo 1337867869 begin to download photo : 1337867868 download finish for photo 1337867495 begin to download photo : 1337867218 download finish for photo 1337867566 begin to download photo : 1337867564 download finish for photo 1337867590 begin to download photo : 1337867498 download finish for photo 1337867868 begin to download photo : 1337867821 download finish for photo 1337867564 begin to download photo : 1337867545 download finish for photo 1337867498 begin to download photo : 1337867925 download finish for photo 1337867218 begin to download photo : 1337867173 download finish for photo 1337867821 begin to download photo : 1337867820 download finish for photo 1337867545 begin to download photo : 1337867544 download finish for photo 1337867173 begin to download photo : 1337866950 download finish for photo 1337867925 begin to download photo : 1337867902 download finish for photo 1337867820 download finish for photo 1337866950 download finish for photo 1337867902 download finish for photo 1337867544 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20586312_15-02-2025_12_08_02.pdf results_Auto_P20586312_15-02-2025_12_08_02.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20586312_15-02-2025_12_08_02.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20586312','results_Auto_P20586312_15-02-2025_12_08_02.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20586312_15-02-2025_12_08_02.pdf','pdf','','0.27','0.0423778629895155') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1337867925, 1337867903, 1337867902, 1337867885, 1337867869, 1337867868, 1337867821, 1337867820, 1337867590, 1337867589] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867925', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867903', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867902', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867885', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867869', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867868', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867821', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867820', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867590', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867589', None, None, None, None, None, '2590787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.013779640197753906 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.1017391681671143 time spend to save output : 0.014202356338500977 total time spend for step 10 : 3.1159415245056152 step11:split_time_score Sat Feb 15 12:08:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('11', 29),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 15022025 20586312 Nombre de photos uploadées : 29 / 23040 (0%) 15022025 20586312 Nombre de photos taguées (types de déchets): 0 / 29 (0%) 15022025 20586312 Nombre de photos taguées (volume) : 0 / 29 (0%) elapsed_time : load_data_split_time_score 3.0994415283203125e-06 elapsed_time : order_list_meta_photo_and_scores 9.5367431640625e-06 ????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0018200874328613281 elapsed_time : insert_dashboard_record_day_entry 0.026080608367919922 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.028720116064827 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20584271_15-02-2025_08_47_23.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20584271 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20584271 AND mptpi.`type`=3726 To do Qualite : 0.043719052866779004 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20584288_15-02-2025_08_53_33.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20584288 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20584288 AND mptpi.`type`=3726 To do Qualite : 0.030735853628317704 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20584687_15-02-2025_10_51_37.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20584687 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20584687 AND mptpi.`type`=3726 To do Qualite : 0.013138868003880314 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20584688_15-02-2025_09_41_09.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20584688 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20584688 AND mptpi.`type`=3726 To do Qualite : 0.03504505734350302 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20584689_15-02-2025_09_39_03.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20584689 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20584689 AND mptpi.`type`=3726 To do Qualite : 0.036175256183249 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20585110_15-02-2025_10_21_25.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20585110 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20585110 AND mptpi.`type`=3726 To do Qualite : 0.02236483073487394 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20585111_15-02-2025_11_31_18.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20585111 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20585111 AND mptpi.`type`=3726 To do Qualite : 0.021583164343394735 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20585342_15-02-2025_10_35_15.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20585342 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20585342 AND mptpi.`type`=3726 To do Qualite : 0.010445117100647125 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20585566_15-02-2025_10_53_15.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20585566 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20585566 AND mptpi.`type`=3726 To do Qualite : 0.012213951064455341 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20585711_15-02-2025_11_11_09.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20585711 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20585711 AND mptpi.`type`=3726 To do Qualite : 0.054081754920421984 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20585838_15-02-2025_11_21_07.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20585838 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20585838 AND mptpi.`type`=3726 To do Qualite : 0.0423778629895155 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20586312_15-02-2025_12_08_02.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20586312 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20586312 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20586313 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20586314 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'15022025': {'nb_upload': 29, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1337867925, 1337867903, 1337867902, 1337867885, 1337867869, 1337867868, 1337867821, 1337867820, 1337867590, 1337867589] Looping around the photos to save general results len do output : 1 /20586312Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867925', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867903', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867902', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867885', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867869', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867868', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867821', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867820', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867590', None, None, None, None, None, '2590787') ('3459', None, None, None, None, None, None, None, '2590787') ('3459', '20586312', '1337867589', None, None, None, None, None, '2590787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014155149459838867 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.4454522132873535 time spend to save output : 0.014431238174438477 total time spend for step 11 : 2.459883451461792 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 42.61user 27.85system 1:43.85elapsed 67%CPU (0avgtext+0avgdata 2473372maxresident)k 1473328inputs+31880outputs (3423major+2082801minor)pagefaults 0swaps