python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 639867 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 24>10 please execute split_portfolio.py -i 21058641 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2637420'] with mtr_portfolio_ids : ['21058641'] and first list_photo_ids : [] new path : /proc/639867/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.1692452430725098 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Wed Mar 5 11:12:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10814 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-05 11:12:31.623234: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-05 11:12:31.651171: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-05 11:12:31.652868: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f1980000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-05 11:12:31.652893: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-05 11:12:31.655591: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-05 11:12:31.880321: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x15902db0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-05 11:12:31.880391: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-05 11:12:31.881890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 11:12:31.882355: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 11:12:31.885605: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 11:12:31.888570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 11:12:31.888964: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 11:12:31.891767: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 11:12:31.893090: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 11:12:31.898352: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 11:12:31.899862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 11:12:31.899940: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 11:12:31.900702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 11:12:31.900717: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 11:12:31.900726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 11:12:31.902012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10023 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-05 11:12:32.163289: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 11:12:32.163399: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 11:12:32.163421: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 11:12:32.163439: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 11:12:32.163458: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 11:12:32.163476: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 11:12:32.163494: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 11:12:32.163512: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 11:12:32.165108: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 11:12:32.166551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 11:12:32.166619: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 11:12:32.166641: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 11:12:32.166662: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 11:12:32.166682: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 11:12:32.166702: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 11:12:32.166724: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 11:12:32.166745: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 11:12:32.168251: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 11:12:32.168282: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 11:12:32.168291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 11:12:32.168299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 11:12:32.169590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10023 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-05 11:12:41.608778: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 11:12:41.796866: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 9 Detection mask done ! Trying to reset tf kernel 640433 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5525 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10814 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.018065690994262695 nb_pixel_total : 1251159 time to create 1 rle with new method : 0.03927469253540039 length of segment : 1600 time for calcul the mask position with numpy : 0.003155946731567383 nb_pixel_total : 301749 time to create 1 rle with new method : 0.007399320602416992 length of segment : 975 time for calcul the mask position with numpy : 0.0003807544708251953 nb_pixel_total : 24991 time to create 1 rle with old method : 0.0273287296295166 length of segment : 320 time for calcul the mask position with numpy : 0.0015461444854736328 nb_pixel_total : 109634 time to create 1 rle with old method : 0.12159538269042969 length of segment : 590 time for calcul the mask position with numpy : 0.00837850570678711 nb_pixel_total : 640005 time to create 1 rle with new method : 0.024997472763061523 length of segment : 1196 time for calcul the mask position with numpy : 0.0031571388244628906 nb_pixel_total : 309594 time to create 1 rle with new method : 0.010295867919921875 length of segment : 929 time for calcul the mask position with numpy : 0.0009877681732177734 nb_pixel_total : 74679 time to create 1 rle with old method : 0.08669567108154297 length of segment : 272 time for calcul the mask position with numpy : 0.002722501754760742 nb_pixel_total : 106908 time to create 1 rle with old method : 0.11930465698242188 length of segment : 956 time for calcul the mask position with numpy : 0.0013382434844970703 nb_pixel_total : 107105 time to create 1 rle with old method : 0.13672399520874023 length of segment : 331 time for calcul the mask position with numpy : 0.00026679039001464844 nb_pixel_total : 13536 time to create 1 rle with old method : 0.017030954360961914 length of segment : 147 time for calcul the mask position with numpy : 0.006268739700317383 nb_pixel_total : 379816 time to create 1 rle with new method : 0.01563429832458496 length of segment : 1748 time for calcul the mask position with numpy : 0.0006527900695800781 nb_pixel_total : 30271 time to create 1 rle with old method : 0.03513145446777344 length of segment : 420 time for calcul the mask position with numpy : 6.914138793945312e-05 nb_pixel_total : 1804 time to create 1 rle with old method : 0.002440214157104492 length of segment : 44 time for calcul the mask position with numpy : 0.0006349086761474609 nb_pixel_total : 28458 time to create 1 rle with old method : 0.03401041030883789 length of segment : 480 time for calcul the mask position with numpy : 0.0002410411834716797 nb_pixel_total : 7279 time to create 1 rle with old method : 0.008800745010375977 length of segment : 112 time for calcul the mask position with numpy : 0.0016765594482421875 nb_pixel_total : 88040 time to create 1 rle with old method : 0.09769058227539062 length of segment : 487 time for calcul the mask position with numpy : 0.002303600311279297 nb_pixel_total : 136978 time to create 1 rle with old method : 0.15509748458862305 length of segment : 503 time for calcul the mask position with numpy : 0.006399393081665039 nb_pixel_total : 430408 time to create 1 rle with new method : 0.015478849411010742 length of segment : 1461 time for calcul the mask position with numpy : 0.00023031234741210938 nb_pixel_total : 7886 time to create 1 rle with old method : 0.010439395904541016 length of segment : 78 time for calcul the mask position with numpy : 0.0011746883392333984 nb_pixel_total : 46725 time to create 1 rle with old method : 0.055457115173339844 length of segment : 295 time for calcul the mask position with numpy : 0.00030303001403808594 nb_pixel_total : 3778 time to create 1 rle with old method : 0.006045341491699219 length of segment : 71 time for calcul the mask position with numpy : 0.0016181468963623047 nb_pixel_total : 36897 time to create 1 rle with old method : 0.05225682258605957 length of segment : 337 time for calcul the mask position with numpy : 0.011813879013061523 nb_pixel_total : 490329 time to create 1 rle with new method : 0.019283294677734375 length of segment : 1188 time for calcul the mask position with numpy : 0.0007061958312988281 nb_pixel_total : 50524 time to create 1 rle with old method : 0.056626319885253906 length of segment : 395 time for calcul the mask position with numpy : 0.0005238056182861328 nb_pixel_total : 27831 time to create 1 rle with old method : 0.030743837356567383 length of segment : 254 time for calcul the mask position with numpy : 0.0005273818969726562 nb_pixel_total : 37721 time to create 1 rle with old method : 0.0431668758392334 length of segment : 310 time for calcul the mask position with numpy : 0.00641179084777832 nb_pixel_total : 381919 time to create 1 rle with new method : 0.01393437385559082 length of segment : 1844 time for calcul the mask position with numpy : 0.0010447502136230469 nb_pixel_total : 55949 time to create 1 rle with old method : 0.06346940994262695 length of segment : 553 time for calcul the mask position with numpy : 0.0002961158752441406 nb_pixel_total : 15978 time to create 1 rle with old method : 0.019498586654663086 length of segment : 208 time for calcul the mask position with numpy : 0.00044536590576171875 nb_pixel_total : 15966 time to create 1 rle with old method : 0.020836830139160156 length of segment : 287 time for calcul the mask position with numpy : 0.0021741390228271484 nb_pixel_total : 100191 time to create 1 rle with old method : 0.11602163314819336 length of segment : 329 time for calcul the mask position with numpy : 0.0009772777557373047 nb_pixel_total : 29483 time to create 1 rle with old method : 0.03600883483886719 length of segment : 233 time for calcul the mask position with numpy : 0.00015497207641601562 nb_pixel_total : 3410 time to create 1 rle with old method : 0.004143238067626953 length of segment : 52 time for calcul the mask position with numpy : 0.014255523681640625 nb_pixel_total : 636924 time to create 1 rle with new method : 0.032601356506347656 length of segment : 1128 time for calcul the mask position with numpy : 0.0007510185241699219 nb_pixel_total : 48717 time to create 1 rle with old method : 0.058151960372924805 length of segment : 548 time for calcul the mask position with numpy : 0.0012352466583251953 nb_pixel_total : 66837 time to create 1 rle with old method : 0.08098983764648438 length of segment : 323 time for calcul the mask position with numpy : 0.0019407272338867188 nb_pixel_total : 166220 time to create 1 rle with new method : 0.004427194595336914 length of segment : 584 time for calcul the mask position with numpy : 6.341934204101562e-05 nb_pixel_total : 634 time to create 1 rle with old method : 0.0008234977722167969 length of segment : 56 time for calcul the mask position with numpy : 0.0007078647613525391 nb_pixel_total : 35210 time to create 1 rle with old method : 0.04341602325439453 length of segment : 200 time for calcul the mask position with numpy : 0.00477290153503418 nb_pixel_total : 281527 time to create 1 rle with new method : 0.011691093444824219 length of segment : 985 time for calcul the mask position with numpy : 0.0002334117889404297 nb_pixel_total : 7608 time to create 1 rle with old method : 0.009428977966308594 length of segment : 126 time for calcul the mask position with numpy : 0.00011587142944335938 nb_pixel_total : 805 time to create 1 rle with old method : 0.001028299331665039 length of segment : 51 time for calcul the mask position with numpy : 0.00043654441833496094 nb_pixel_total : 13444 time to create 1 rle with old method : 0.01612997055053711 length of segment : 190 time for calcul the mask position with numpy : 0.0021576881408691406 nb_pixel_total : 127044 time to create 1 rle with old method : 0.14583921432495117 length of segment : 506 time for calcul the mask position with numpy : 0.0059528350830078125 nb_pixel_total : 365946 time to create 1 rle with new method : 0.01701641082763672 length of segment : 1340 time for calcul the mask position with numpy : 0.004491567611694336 nb_pixel_total : 331661 time to create 1 rle with new method : 0.011690378189086914 length of segment : 1347 time for calcul the mask position with numpy : 0.0011584758758544922 nb_pixel_total : 44736 time to create 1 rle with old method : 0.056714534759521484 length of segment : 339 time for calcul the mask position with numpy : 0.005579471588134766 nb_pixel_total : 39758 time to create 1 rle with old method : 0.054369449615478516 length of segment : 283 time for calcul the mask position with numpy : 0.0005233287811279297 nb_pixel_total : 22523 time to create 1 rle with old method : 0.0277402400970459 length of segment : 154 time for calcul the mask position with numpy : 0.005117177963256836 nb_pixel_total : 191239 time to create 1 rle with new method : 0.010024547576904297 length of segment : 1128 time for calcul the mask position with numpy : 0.00010514259338378906 nb_pixel_total : 1225 time to create 1 rle with old method : 0.0016584396362304688 length of segment : 45 time for calcul the mask position with numpy : 0.008433818817138672 nb_pixel_total : 303802 time to create 1 rle with new method : 0.025243520736694336 length of segment : 1096 time for calcul the mask position with numpy : 0.0009315013885498047 nb_pixel_total : 29218 time to create 1 rle with old method : 0.03513956069946289 length of segment : 275 time for calcul the mask position with numpy : 0.00011396408081054688 nb_pixel_total : 1841 time to create 1 rle with old method : 0.002401590347290039 length of segment : 43 time for calcul the mask position with numpy : 0.0022172927856445312 nb_pixel_total : 17189 time to create 1 rle with old method : 0.022495031356811523 length of segment : 319 time spent for convertir_results : 6.28479266166687 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 55 chid ids of type : 3663 Number RLEs to save : 30071 save missing photos in datou_result : time spend for datou_step_exec : 30.13066864013672 time spend to save output : 1.6763715744018555 total time spend for step 1 : 31.807040214538574 step2:crop_condition Wed Mar 5 11:13:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 55 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340990442, 1340990042, 1340990041, 1340989937, 1340989898, 1340988313, 1340988197, 1340988092, 1340988089, 1340987893] Looping around the photos to save general results len do output : 20 /-3694348888Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348887Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348886Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348896Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348909Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348910Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348917Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348930Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348893Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348898Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348905Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348904Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348903Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348916Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348914Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348925Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348912Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348920Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348922Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694348932Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990442', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990042', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990041', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989937', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989898', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988313', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988197', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988092', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988089', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340987893', None, None, None, None, None, '2637420') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 70 time used for this insertion : 0.02135324478149414 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.510682821273804 time spend to save output : 0.022655487060546875 total time spend for step 2 : 6.533338308334351 step3:thcl Wed Mar 5 11:13:07 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.006199359893798828 time to convert the images to numpy array : 0.00460052490234375 time to import caffe and check if the image exist : 0.007165431976318359 time to convert the images to numpy array : 0.008794546127319336 time to import caffe and check if the image exist : 0.0037126541137695312 time to convert the images to numpy array : 0.014809131622314453 time to import caffe and check if the image exist : 0.006031990051269531 time to convert the images to numpy array : 0.016405105590820312 time to import caffe and check if the image exist : 0.0051500797271728516 time to convert the images to numpy array : 0.017968416213989258 time to import caffe and check if the image exist : 0.005357980728149414 time to convert the images to numpy array : 0.022592544555664062 time to import caffe and check if the image exist : 0.0050351619720458984 time to convert the images to numpy array : 0.027719736099243164 time to import caffe and check if the image exist : 0.006991863250732422 time to convert the images to numpy array : 0.02806544303894043 time to import caffe and check if the image exist : 0.004081010818481445 time to convert the images to numpy array : 0.03241586685180664 time to import caffe and check if the image exist : 0.005202054977416992 time to convert the images to numpy array : 0.043332576751708984 total time to convert the images to numpy array : 0.24852681159973145 list photo_ids error: [] list photo_ids correct : [-3694348917, -3694348930, -3694348912, -3694348920, -3694348893, -3694348898, -3694348922, -3694348932, -3694348905, -3694348904, -3694348909, -3694348910, -3694348903, -3694348916, -3694348886, -3694348896, -3694348888, -3694348887, -3694348914, -3694348925] number of photos to traite : 20 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10814 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10595 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.24381446838378906 time used to do the prediction : 0.0848550796508789 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10593 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10593 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.2310326099395752 time used to do the prediction : 0.04924273490905762 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.0265579223632812e-05 save missing photos in datou_result : time spend for datou_step_exec : 7.671363592147827 time spend to save output : 0.0004143714904785156 total time spend for step 3 : 7.671777963638306 step4:merge_mask_thcl_custom Wed Mar 5 11:13:14 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 55 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1340990442, 1340990042, 1340990041, 1340989937, 1340989898, 1340988313, 1340988197, 1340988092, 1340988089, 1340987893] Looping around the photos to save general results len do output : 10 /1340990442Didn't retrieve data .Didn't retrieve data . /1340990042Didn't retrieve data .Didn't retrieve data . /1340990041Didn't retrieve data .Didn't retrieve data . /1340989937Didn't retrieve data .Didn't retrieve data . /1340989898Didn't retrieve data .Didn't retrieve data . /1340988313Didn't retrieve data .Didn't retrieve data . /1340988197Didn't retrieve data .Didn't retrieve data . /1340988092Didn't retrieve data .Didn't retrieve data . /1340988089Didn't retrieve data .Didn't retrieve data . /1340987893Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990442', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990042', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990041', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989937', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989898', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988313', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988197', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988092', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988089', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340987893', None, None, None, None, None, '2637420') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014415264129638672 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.0538630485534668 time spend to save output : 0.015227079391479492 total time spend for step 4 : 0.06909012794494629 step5:rle_unique_nms_with_priority Wed Mar 5 11:13:14 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 0.5761866569519043 time for calcul the mask position with numpy : 0.011258125305175781 nb_pixel_total : 720604 time to create 1 rle with new method : 0.4700748920440674 time for calcul the mask position with numpy : 0.008095979690551758 nb_pixel_total : 103071 time to create 1 rle with old method : 0.13405060768127441 time for calcul the mask position with numpy : 0.04992389678955078 nb_pixel_total : 1249925 time to create 1 rle with new method : 0.33948397636413574 create new chi : 1.029409646987915 time to delete rle : 0.0294649600982666 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 5174 TO DO : save crop sub photo not yet done ! save time : 0.3189516067504883 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.9414002895355225 time for calcul the mask position with numpy : 0.20691823959350586 nb_pixel_total : 1215585 time to create 1 rle with new method : 0.07529497146606445 time for calcul the mask position with numpy : 0.0065670013427734375 nb_pixel_total : 78115 time to create 1 rle with old method : 0.09240198135375977 time for calcul the mask position with numpy : 0.012331247329711914 nb_pixel_total : 645473 time to create 1 rle with new method : 0.2870142459869385 time for calcul the mask position with numpy : 0.007269859313964844 nb_pixel_total : 109488 time to create 1 rle with old method : 0.12526273727416992 time for calcul the mask position with numpy : 0.010371208190917969 nb_pixel_total : 24939 time to create 1 rle with old method : 0.030602455139160156 create new chi : 0.8687243461608887 time to delete rle : 0.0006856918334960938 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 5750 TO DO : save crop sub photo not yet done ! save time : 0.36542582511901855 nb_obj : 8 nb_hashtags : 2 time to prepare the origin masks : 0.9897346496582031 time for calcul the mask position with numpy : 0.04126620292663574 nb_pixel_total : 1370220 time to create 1 rle with new method : 0.35476040840148926 time for calcul the mask position with numpy : 0.006849050521850586 nb_pixel_total : 106874 time to create 1 rle with old method : 0.11761736869812012 time for calcul the mask position with numpy : 0.006448268890380859 nb_pixel_total : 1791 time to create 1 rle with old method : 0.002057313919067383 time for calcul the mask position with numpy : 0.006282806396484375 nb_pixel_total : 3458 time to create 1 rle with old method : 0.004444122314453125 time for calcul the mask position with numpy : 0.0067386627197265625 nb_pixel_total : 74206 time to create 1 rle with old method : 0.08743524551391602 time for calcul the mask position with numpy : 0.009100198745727539 nb_pixel_total : 378740 time to create 1 rle with new method : 0.07742691040039062 time for calcul the mask position with numpy : 0.006231546401977539 nb_pixel_total : 2593 time to create 1 rle with old method : 0.003190279006958008 time for calcul the mask position with numpy : 0.005803585052490234 nb_pixel_total : 28610 time to create 1 rle with old method : 0.0325474739074707 time for calcul the mask position with numpy : 0.0067255496978759766 nb_pixel_total : 107108 time to create 1 rle with old method : 0.12151741981506348 create new chi : 0.9112730026245117 time to delete rle : 0.0008749961853027344 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 9152 TO DO : save crop sub photo not yet done ! save time : 0.5620331764221191 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 0.4226686954498291 time for calcul the mask position with numpy : 0.018181562423706055 nb_pixel_total : 1404237 time to create 1 rle with new method : 0.42508697509765625 time for calcul the mask position with numpy : 0.007019519805908203 nb_pixel_total : 136838 time to create 1 rle with old method : 0.151017427444458 time for calcul the mask position with numpy : 0.008422613143920898 nb_pixel_total : 429483 time to create 1 rle with new method : 0.3029444217681885 time for calcul the mask position with numpy : 0.006411075592041016 nb_pixel_total : 87957 time to create 1 rle with old method : 0.10071706771850586 time for calcul the mask position with numpy : 0.006117105484008789 nb_pixel_total : 7828 time to create 1 rle with old method : 0.009099960327148438 time for calcul the mask position with numpy : 0.0060672760009765625 nb_pixel_total : 7257 time to create 1 rle with old method : 0.008368253707885742 create new chi : 1.057164192199707 time to delete rle : 0.0006139278411865234 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6182 TO DO : save crop sub photo not yet done ! save time : 0.45351552963256836 nb_obj : 7 nb_hashtags : 2 time to prepare the origin masks : 0.7762444019317627 time for calcul the mask position with numpy : 0.33887577056884766 nb_pixel_total : 1403095 time to create 1 rle with new method : 0.11059880256652832 time for calcul the mask position with numpy : 0.010665416717529297 nb_pixel_total : 502998 time to create 1 rle with new method : 0.3552510738372803 time for calcul the mask position with numpy : 0.006430625915527344 nb_pixel_total : 1245 time to create 1 rle with old method : 0.0016355514526367188 time for calcul the mask position with numpy : 0.006306648254394531 nb_pixel_total : 3753 time to create 1 rle with old method : 0.004433631896972656 time for calcul the mask position with numpy : 0.007668256759643555 nb_pixel_total : 46609 time to create 1 rle with old method : 0.05695343017578125 time for calcul the mask position with numpy : 0.006929874420166016 nb_pixel_total : 37613 time to create 1 rle with old method : 0.04339861869812012 time for calcul the mask position with numpy : 0.006714344024658203 nb_pixel_total : 27774 time to create 1 rle with old method : 0.032578468322753906 time for calcul the mask position with numpy : 0.007359504699707031 nb_pixel_total : 50513 time to create 1 rle with old method : 0.06260061264038086 create new chi : 1.075739860534668 time to delete rle : 0.0007345676422119141 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6064 TO DO : save crop sub photo not yet done ! save time : 0.3923923969268799 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.4123551845550537 time for calcul the mask position with numpy : 0.027136564254760742 nb_pixel_total : 1580928 time to create 1 rle with new method : 0.33270812034606934 time for calcul the mask position with numpy : 0.008039712905883789 nb_pixel_total : 15942 time to create 1 rle with old method : 0.017601728439331055 time for calcul the mask position with numpy : 0.00666499137878418 nb_pixel_total : 15821 time to create 1 rle with old method : 0.023617267608642578 time for calcul the mask position with numpy : 0.013181686401367188 nb_pixel_total : 55883 time to create 1 rle with old method : 0.05915093421936035 time for calcul the mask position with numpy : 0.01361989974975586 nb_pixel_total : 405026 time to create 1 rle with new method : 0.04316282272338867 create new chi : 0.5531742572784424 time to delete rle : 0.0006823539733886719 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 6106 TO DO : save crop sub photo not yet done ! save time : 0.3509066104888916 nb_obj : 8 nb_hashtags : 4 time to prepare the origin masks : 1.2551910877227783 time for calcul the mask position with numpy : 0.1626441478729248 nb_pixel_total : 1083516 time to create 1 rle with new method : 0.09780430793762207 time for calcul the mask position with numpy : 0.007071971893310547 nb_pixel_total : 3402 time to create 1 rle with old method : 0.003891468048095703 time for calcul the mask position with numpy : 0.007176876068115234 nb_pixel_total : 99095 time to create 1 rle with old method : 0.11221480369567871 time for calcul the mask position with numpy : 0.006819725036621094 nb_pixel_total : 28915 time to create 1 rle with old method : 0.03193473815917969 time for calcul the mask position with numpy : 0.006753683090209961 nb_pixel_total : 27260 time to create 1 rle with old method : 0.03062582015991211 time for calcul the mask position with numpy : 0.007052183151245117 nb_pixel_total : 630 time to create 1 rle with old method : 0.0007302761077880859 time for calcul the mask position with numpy : 0.007778167724609375 nb_pixel_total : 165961 time to create 1 rle with new method : 0.11486244201660156 time for calcul the mask position with numpy : 0.0177154541015625 nb_pixel_total : 635399 time to create 1 rle with new method : 0.4045991897583008 time for calcul the mask position with numpy : 0.006923675537109375 nb_pixel_total : 29422 time to create 1 rle with old method : 0.032811641693115234 create new chi : 1.085993766784668 time to delete rle : 0.0008821487426757812 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 7067 TO DO : save crop sub photo not yet done ! save time : 0.40183138847351074 nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.34484314918518066 time for calcul the mask position with numpy : 0.029659032821655273 nb_pixel_total : 1778452 time to create 1 rle with new method : 0.04077029228210449 time for calcul the mask position with numpy : 0.008143186569213867 nb_pixel_total : 799 time to create 1 rle with old method : 0.0009412765502929688 time for calcul the mask position with numpy : 0.008687973022460938 nb_pixel_total : 251674 time to create 1 rle with new method : 0.03716254234313965 time for calcul the mask position with numpy : 0.0070171356201171875 nb_pixel_total : 35103 time to create 1 rle with old method : 0.04406094551086426 time for calcul the mask position with numpy : 0.006965160369873047 nb_pixel_total : 7572 time to create 1 rle with old method : 0.008154869079589844 create new chi : 0.1924130916595459 time to delete rle : 0.0004520416259765625 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 3626 TO DO : save crop sub photo not yet done ! save time : 0.24202561378479004 nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.44779467582702637 time for calcul the mask position with numpy : 0.025661230087280273 nb_pixel_total : 1635362 time to create 1 rle with new method : 0.036714792251586914 time for calcul the mask position with numpy : 0.006941318511962891 nb_pixel_total : 47695 time to create 1 rle with old method : 0.05217170715332031 time for calcul the mask position with numpy : 0.008232831954956055 nb_pixel_total : 250228 time to create 1 rle with new method : 0.037693023681640625 time for calcul the mask position with numpy : 0.007714033126831055 nb_pixel_total : 126918 time to create 1 rle with old method : 0.1479787826538086 time for calcul the mask position with numpy : 0.007075309753417969 nb_pixel_total : 13397 time to create 1 rle with old method : 0.015200376510620117 create new chi : 0.35247015953063965 time to delete rle : 0.0005524158477783203 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 4656 TO DO : save crop sub photo not yet done ! save time : 0.28151607513427734 nb_obj : 8 nb_hashtags : 3 time to prepare the origin masks : 0.7616090774536133 time for calcul the mask position with numpy : 0.15741801261901855 nb_pixel_total : 1532253 time to create 1 rle with new method : 0.33321666717529297 time for calcul the mask position with numpy : 0.006653785705566406 nb_pixel_total : 29118 time to create 1 rle with old method : 0.03487372398376465 time for calcul the mask position with numpy : 0.007090568542480469 nb_pixel_total : 1217 time to create 1 rle with old method : 0.0014336109161376953 time for calcul the mask position with numpy : 0.0061376094818115234 nb_pixel_total : 1811 time to create 1 rle with old method : 0.0021572113037109375 time for calcul the mask position with numpy : 0.007971763610839844 nb_pixel_total : 206103 time to create 1 rle with new method : 0.0832664966583252 time for calcul the mask position with numpy : 0.009181737899780273 nb_pixel_total : 196393 time to create 1 rle with new method : 0.4153923988342285 time for calcul the mask position with numpy : 0.006789445877075195 nb_pixel_total : 22338 time to create 1 rle with old method : 0.03914642333984375 time for calcul the mask position with numpy : 0.0074443817138671875 nb_pixel_total : 39673 time to create 1 rle with old method : 0.04548478126525879 time for calcul the mask position with numpy : 0.0074615478515625 nb_pixel_total : 44694 time to create 1 rle with old method : 0.05207467079162598 create new chi : 1.2482004165649414 time to delete rle : 0.0007550716400146484 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 6131 TO DO : save crop sub photo not yet done ! save time : 0.3724963665008545 map_output_result : {1340990442: (0.0, 'Should be the crop_list due to order', 0), 1340990042: (0.0, 'Should be the crop_list due to order', 0), 1340990041: (0.0, 'Should be the crop_list due to order', 0), 1340989937: (0.0, 'Should be the crop_list due to order', 0), 1340989898: (0.0, 'Should be the crop_list due to order', 0), 1340988313: (0.0, 'Should be the crop_list due to order', 0), 1340988197: (0.0, 'Should be the crop_list due to order', 0), 1340988092: (0.0, 'Should be the crop_list due to order', 0), 1340988089: (0.0, 'Should be the crop_list due to order', 0), 1340987893: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1340990442, 1340990042, 1340990041, 1340989937, 1340989898, 1340988313, 1340988197, 1340988092, 1340988089, 1340987893] Looping around the photos to save general results len do output : 10 /1340990442.Didn't retrieve data . /1340990042.Didn't retrieve data . /1340990041.Didn't retrieve data . /1340989937.Didn't retrieve data . /1340989898.Didn't retrieve data . /1340988313.Didn't retrieve data . /1340988197.Didn't retrieve data . /1340988092.Didn't retrieve data . /1340988089.Didn't retrieve data . /1340987893.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990442', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990042', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990041', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989937', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989898', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988313', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988197', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988092', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988089', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340987893', None, None, None, None, None, '2637420') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013164997100830078 save_final save missing photos in datou_result : time spend for datou_step_exec : 19.394622564315796 time spend to save output : 0.013672113418579102 total time spend for step 5 : 19.408294677734375 step6:crop_condition Wed Mar 5 11:13:34 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 64 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741169615_639867 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.736088752746582 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741169616_639867 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5647199153900146 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741169617_639867 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6387159824371338 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340990442, 1340990042, 1340990041, 1340989937, 1340989898, 1340988313, 1340988197, 1340988092, 1340988089, 1340987893] Looping around the photos to save general results len do output : 3 /1340993532Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340993533Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340993535Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990442', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990042', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990041', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989937', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989898', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988313', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988197', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988092', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988089', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340987893', None, None, None, None, None, '2637420') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 19 time used for this insertion : 0.021080493927001953 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.6981372833251953 time spend to save output : 0.02154994010925293 total time spend for step 6 : 3.7196872234344482 step7:ventilate_hashtags_in_portfolio Wed Mar 5 11:13:38 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 21058641 get user id for portfolio 21058641 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21058641 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Teint_Dans_La_Masse','autre_refus','mal_croppe','kraft','flou','metal','plastique','Carton_brun','papier','cartonnette','environnement','Carton_gris')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21058641 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Teint_Dans_La_Masse','autre_refus','mal_croppe','kraft','flou','metal','plastique','Carton_brun','papier','cartonnette','environnement','Carton_gris')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/21058966,21058967,21058968,21058969,21058970,21058971,21058972,21058973,21058974,21058975,21058976,21058977?tags=mal_croppe,Carton_gris,plastique,Carton_brun,flou,autre_refus,Teint_Dans_La_Masse,kraft,metal,papier,cartonnette,environnement Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1340990442, 1340990042, 1340990041, 1340989937, 1340989898, 1340988313, 1340988197, 1340988092, 1340988089, 1340987893] Looping around the photos to save general results len do output : 1 /21058641. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990442', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990042', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990041', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989937', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989898', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988313', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988197', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988092', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988089', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340987893', None, None, None, None, None, '2637420') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013791084289550781 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.412780523300171 time spend to save output : 0.014129877090454102 total time spend for step 7 : 1.426910400390625 step8:final Wed Mar 5 11:13:39 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1340990442: ('0.01736342564419555',), 1340990042: ('0.01736342564419555',), 1340990041: ('0.01736342564419555',), 1340989937: ('0.01736342564419555',), 1340989898: ('0.01736342564419555',), 1340988313: ('0.01736342564419555',), 1340988197: ('0.01736342564419555',), 1340988092: ('0.01736342564419555',), 1340988089: ('0.01736342564419555',), 1340987893: ('0.01736342564419555',)} new output for save of step final : {1340990442: ('0.01736342564419555',), 1340990042: ('0.01736342564419555',), 1340990041: ('0.01736342564419555',), 1340989937: ('0.01736342564419555',), 1340989898: ('0.01736342564419555',), 1340988313: ('0.01736342564419555',), 1340988197: ('0.01736342564419555',), 1340988092: ('0.01736342564419555',), 1340988089: ('0.01736342564419555',), 1340987893: ('0.01736342564419555',)} [1340990442, 1340990042, 1340990041, 1340989937, 1340989898, 1340988313, 1340988197, 1340988092, 1340988089, 1340987893] Looping around the photos to save general results len do output : 10 /1340990442.Didn't retrieve data . /1340990042.Didn't retrieve data . /1340990041.Didn't retrieve data . /1340989937.Didn't retrieve data . /1340989898.Didn't retrieve data . /1340988313.Didn't retrieve data . /1340988197.Didn't retrieve data . /1340988092.Didn't retrieve data . /1340988089.Didn't retrieve data . /1340987893.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990442', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990042', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990041', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989937', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989898', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988313', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988197', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988092', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988089', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340987893', None, None, None, None, None, '2637420') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.012724637985229492 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10289502143859863 time spend to save output : 0.013268470764160156 total time spend for step 8 : 0.11616349220275879 step9:velours_tree Wed Mar 5 11:13:39 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.03195786476135254 time spend to save output : 4.9591064453125e-05 total time spend for step 9 : 0.032007455825805664 step10:send_mail_cod Wed Mar 5 11:13:39 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21058641_05-03-2025_11_13_39.pdf 21058966 imagette210589661741169619 21058968 change filename to text .imagette210589681741169619 21058970 imagette210589701741169619 21058971 change filename to text .imagette210589711741169619 21058973 change filename to text .imagette210589731741169619 21058974 imagette210589741741169619 21058976 imagette210589761741169619 21058967 imagette210589671741169619 21058969 change filename to text .imagette210589691741169619 21058972 imagette210589721741169619 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21058641 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21058966,21058967,21058968,21058969,21058970,21058971,21058972,21058973,21058974,21058975,21058976,21058977?tags=mal_croppe,Carton_gris,plastique,Carton_brun,flou,autre_refus,Teint_Dans_La_Masse,kraft,metal,papier,cartonnette,environnement your option no_mail is active, we will not send the real mail to your client args[1340990442] : ('0.01736342564419555',) no score found for photo 1340990442 We are sending mail with results at report@fotonower.com args[1340990042] : ('0.01736342564419555',) no score found for photo 1340990042 We are sending mail with results at report@fotonower.com args[1340990041] : ('0.01736342564419555',) no score found for photo 1340990041 We are sending mail with results at report@fotonower.com args[1340989937] : ('0.01736342564419555',) no score found for photo 1340989937 We are sending mail with results at report@fotonower.com args[1340989898] : ('0.01736342564419555',) no score found for photo 1340989898 We are sending mail with results at report@fotonower.com args[1340988313] : ('0.01736342564419555',) no score found for photo 1340988313 We are sending mail with results at report@fotonower.com args[1340988197] : ('0.01736342564419555',) no score found for photo 1340988197 We are sending mail with results at report@fotonower.com args[1340988092] : ('0.01736342564419555',) no score found for photo 1340988092 We are sending mail with results at report@fotonower.com args[1340988089] : ('0.01736342564419555',) no score found for photo 1340988089 We are sending mail with results at report@fotonower.com args[1340987893] : ('0.01736342564419555',) no score found for photo 1340987893 We are sending mail with results at report@fotonower.com refus_total : 0.01736342564419555 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=21058641 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1340988089,1340988092,1340989898,1340991384,1340991380,1340991318,1340991314,1340990573,1340990442,1340990042,1340990041,1340989937,1340987652,1340988313,1340988197,1340987893,1340987890,1340987735,1340987729) Found this number of photos: 19 begin to download photo : 1340988089 begin to download photo : 1340991380 begin to download photo : 1340990442 begin to download photo : 1340987652 begin to download photo : 1340987890 download finish for photo 1340991380 begin to download photo : 1340991318 download finish for photo 1340988089 begin to download photo : 1340988092 download finish for photo 1340990442 begin to download photo : 1340990042 download finish for photo 1340987652 begin to download photo : 1340988313 download finish for photo 1340987890 begin to download photo : 1340987735 download finish for photo 1340988313 begin to download photo : 1340988197 download finish for photo 1340991318 begin to download photo : 1340991314 download finish for photo 1340990042 begin to download photo : 1340990041 download finish for photo 1340988092 begin to download photo : 1340989898 download finish for photo 1340991314 begin to download photo : 1340990573 download finish for photo 1340987735 begin to download photo : 1340987729 download finish for photo 1340990041 begin to download photo : 1340989937 download finish for photo 1340988197 begin to download photo : 1340987893 download finish for photo 1340989898 begin to download photo : 1340991384 download finish for photo 1340990573 download finish for photo 1340989937 download finish for photo 1340987893 download finish for photo 1340987729 download finish for photo 1340991384 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21058641_05-03-2025_11_13_39.pdf results_Auto_P21058641_05-03-2025_11_13_39.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21058641_05-03-2025_11_13_39.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','21058641','results_Auto_P21058641_05-03-2025_11_13_39.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21058641_05-03-2025_11_13_39.pdf','pdf','','0.16','0.01736342564419555') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1340990442, 1340990042, 1340990041, 1340989937, 1340989898, 1340988313, 1340988197, 1340988092, 1340988089, 1340987893] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990442', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990042', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990041', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989937', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989898', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988313', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988197', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988092', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988089', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340987893', None, None, None, None, None, '2637420') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.011908292770385742 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.970670223236084 time spend to save output : 0.012117862701416016 total time spend for step 10 : 1.9827880859375 step11:split_time_score Wed Mar 5 11:13:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 24),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 05032025 21058641 Nombre de photos uploadées : 24 / 23040 (0%) 05032025 21058641 Nombre de photos taguées (types de déchets): 0 / 24 (0%) 05032025 21058641 Nombre de photos taguées (volume) : 0 / 24 (0%) elapsed_time : load_data_split_time_score 6.4373016357421875e-06 elapsed_time : order_list_meta_photo_and_scores 1.5735626220703125e-05 ???????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0012960433959960938 elapsed_time : insert_dashboard_record_day_entry 0.024453163146972656 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.02711167109232465 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21051474_05-03-2025_08_21_34.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21051474 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21051474 AND mptpi.`type`=3726 To do Qualite : 0.015165121742086377 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21051475_05-03-2025_08_17_44.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21051475 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21051475 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21051476 order by id desc limit 1 Qualite : 0.04583871416776438 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21051477_05-03-2025_08_11_04.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21051477 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21051477 AND mptpi.`type`=3726 To do Qualite : 0.04707189105017095 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21052624_05-03-2025_08_37_46.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21052624 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21052624 AND mptpi.`type`=3726 To do Qualite : 0.02047683764150151 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21053151_05-03-2025_08_47_09.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21053151 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21053151 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21058636 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21058637 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21058638 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21058639 order by id desc limit 1 Qualite : 0.03218183001504117 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21058640_05-03-2025_11_13_29.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21058640 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21058640 AND mptpi.`type`=3726 To do Qualite : 0.014566279631737625 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21058641_05-03-2025_11_13_39.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21058641 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21058641 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'05032025': {'nb_upload': 24, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1340990442, 1340990042, 1340990041, 1340989937, 1340989898, 1340988313, 1340988197, 1340988092, 1340988089, 1340987893] Looping around the photos to save general results len do output : 1 /21058641Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990442', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990042', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340990041', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989937', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340989898', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988313', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988197', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988092', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340988089', None, None, None, None, None, '2637420') ('3459', None, None, None, None, None, None, None, '2637420') ('3459', None, '1340987893', None, None, None, None, None, '2637420') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.012949228286743164 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.645179748535156 time spend to save output : 0.013139724731445312 total time spend for step 11 : 4.658319473266602 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 40.09user 30.35system 1:22.41elapsed 85%CPU (0avgtext+0avgdata 2766252maxresident)k 8656inputs+23088outputs (88major+2032343minor)pagefaults 0swaps