python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3017765 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 22>10 please execute split_portfolio.py -i 20631046 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2595746'] with mtr_portfolio_ids : ['20631046'] and first list_photo_ids : [] new path : /proc/3017765/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.113711357116699 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue Feb 18 10:02:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6619 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-18 10:02:34.367418: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-18 10:02:34.407159: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-18 10:02:34.412046: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe5d4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-18 10:02:34.412128: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-18 10:02:34.419701: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-18 10:02:34.674027: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2b073490 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-18 10:02:34.674091: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-18 10:02:34.675663: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-18 10:02:34.676130: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-18 10:02:34.679462: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-18 10:02:34.697584: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-18 10:02:34.698044: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-18 10:02:34.730638: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-18 10:02:34.735192: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-18 10:02:34.784801: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-18 10:02:34.786106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-18 10:02:34.786513: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-18 10:02:34.787186: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-18 10:02:34.787204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-18 10:02:34.787214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-18 10:02:34.789125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6079 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-18 10:02:35.125737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-18 10:02:35.125908: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-18 10:02:35.125938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-18 10:02:35.125962: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-18 10:02:35.125981: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-18 10:02:35.125998: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-18 10:02:35.126013: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-18 10:02:35.126035: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-18 10:02:35.127132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-18 10:02:35.128407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-18 10:02:35.128453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-18 10:02:35.128476: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-18 10:02:35.128494: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-18 10:02:35.128508: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-18 10:02:35.128526: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-18 10:02:35.128542: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-18 10:02:35.128555: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-18 10:02:35.129760: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-18 10:02:35.129803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-18 10:02:35.129812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-18 10:02:35.129820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-18 10:02:35.130992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6079 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-18 10:02:47.630681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-18 10:02:47.795416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 2.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.60000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 141.60000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 139.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 5.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 143.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 142.60000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 13 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 4.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 139.60000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 139.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 5.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 140.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 142.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 144.60000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 6 Detection mask done ! Trying to reset tf kernel 3018170 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1330 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 6619 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0005066394805908203 nb_pixel_total : 23170 time to create 1 rle with old method : 0.029707908630371094 length of segment : 184 time for calcul the mask position with numpy : 0.0025200843811035156 nb_pixel_total : 161131 time to create 1 rle with new method : 0.006542205810546875 length of segment : 612 time for calcul the mask position with numpy : 0.0005061626434326172 nb_pixel_total : 27033 time to create 1 rle with old method : 0.030739545822143555 length of segment : 241 time for calcul the mask position with numpy : 0.0009558200836181641 nb_pixel_total : 72565 time to create 1 rle with old method : 0.08099102973937988 length of segment : 412 time for calcul the mask position with numpy : 0.0015425682067871094 nb_pixel_total : 59419 time to create 1 rle with old method : 0.07100915908813477 length of segment : 312 time for calcul the mask position with numpy : 0.0007584095001220703 nb_pixel_total : 50306 time to create 1 rle with old method : 0.0580441951751709 length of segment : 480 time for calcul the mask position with numpy : 0.0068395137786865234 nb_pixel_total : 462783 time to create 1 rle with new method : 0.5062663555145264 length of segment : 865 time for calcul the mask position with numpy : 0.00115966796875 nb_pixel_total : 68478 time to create 1 rle with old method : 0.07668519020080566 length of segment : 459 time for calcul the mask position with numpy : 0.0022628307342529297 nb_pixel_total : 172165 time to create 1 rle with new method : 0.005792856216430664 length of segment : 784 time for calcul the mask position with numpy : 0.001753091812133789 nb_pixel_total : 92139 time to create 1 rle with old method : 0.10500788688659668 length of segment : 424 time for calcul the mask position with numpy : 0.0013079643249511719 nb_pixel_total : 88280 time to create 1 rle with old method : 0.09711837768554688 length of segment : 306 time for calcul the mask position with numpy : 0.0013003349304199219 nb_pixel_total : 95817 time to create 1 rle with old method : 0.10972237586975098 length of segment : 405 time for calcul the mask position with numpy : 0.00047469139099121094 nb_pixel_total : 14105 time to create 1 rle with old method : 0.016782283782958984 length of segment : 116 time for calcul the mask position with numpy : 0.0002722740173339844 nb_pixel_total : 14566 time to create 1 rle with old method : 0.016480445861816406 length of segment : 151 time for calcul the mask position with numpy : 0.0003552436828613281 nb_pixel_total : 9948 time to create 1 rle with old method : 0.011571407318115234 length of segment : 153 time for calcul the mask position with numpy : 0.0004048347473144531 nb_pixel_total : 12142 time to create 1 rle with old method : 0.01403188705444336 length of segment : 186 time for calcul the mask position with numpy : 0.002409696578979492 nb_pixel_total : 88983 time to create 1 rle with old method : 0.11723470687866211 length of segment : 609 time for calcul the mask position with numpy : 0.0006120204925537109 nb_pixel_total : 37392 time to create 1 rle with old method : 0.04314160346984863 length of segment : 213 time for calcul the mask position with numpy : 0.004538059234619141 nb_pixel_total : 221312 time to create 1 rle with new method : 0.017729520797729492 length of segment : 653 time for calcul the mask position with numpy : 0.0009300708770751953 nb_pixel_total : 24410 time to create 1 rle with old method : 0.026515483856201172 length of segment : 207 time for calcul the mask position with numpy : 0.004004001617431641 nb_pixel_total : 143256 time to create 1 rle with old method : 0.15143871307373047 length of segment : 521 time for calcul the mask position with numpy : 0.0009818077087402344 nb_pixel_total : 26999 time to create 1 rle with old method : 0.031039953231811523 length of segment : 187 time for calcul the mask position with numpy : 0.0005695819854736328 nb_pixel_total : 12944 time to create 1 rle with old method : 0.015102386474609375 length of segment : 196 time for calcul the mask position with numpy : 0.0029883384704589844 nb_pixel_total : 104482 time to create 1 rle with old method : 0.11618542671203613 length of segment : 501 time for calcul the mask position with numpy : 0.0024390220642089844 nb_pixel_total : 86707 time to create 1 rle with old method : 0.09146499633789062 length of segment : 457 time for calcul the mask position with numpy : 0.001827239990234375 nb_pixel_total : 78816 time to create 1 rle with old method : 0.0887448787689209 length of segment : 335 time for calcul the mask position with numpy : 0.0035080909729003906 nb_pixel_total : 100413 time to create 1 rle with old method : 0.11344051361083984 length of segment : 913 time for calcul the mask position with numpy : 0.0009534358978271484 nb_pixel_total : 24124 time to create 1 rle with old method : 0.026612520217895508 length of segment : 425 time for calcul the mask position with numpy : 0.001383066177368164 nb_pixel_total : 44859 time to create 1 rle with old method : 0.05064558982849121 length of segment : 214 time for calcul the mask position with numpy : 0.04312324523925781 nb_pixel_total : 1717199 time to create 1 rle with new method : 0.2501654624938965 length of segment : 2892 time for calcul the mask position with numpy : 0.02709054946899414 nb_pixel_total : 1185731 time to create 1 rle with new method : 0.21552085876464844 length of segment : 1369 time for calcul the mask position with numpy : 0.0006008148193359375 nb_pixel_total : 11885 time to create 1 rle with old method : 0.013748645782470703 length of segment : 174 time for calcul the mask position with numpy : 0.0028994083404541016 nb_pixel_total : 153006 time to create 1 rle with new method : 0.0048351287841796875 length of segment : 459 time for calcul the mask position with numpy : 0.0015954971313476562 nb_pixel_total : 85524 time to create 1 rle with old method : 0.09374213218688965 length of segment : 353 time for calcul the mask position with numpy : 0.00043511390686035156 nb_pixel_total : 12029 time to create 1 rle with old method : 0.013872861862182617 length of segment : 157 time for calcul the mask position with numpy : 0.0017192363739013672 nb_pixel_total : 93472 time to create 1 rle with old method : 0.10292911529541016 length of segment : 366 time for calcul the mask position with numpy : 0.0007407665252685547 nb_pixel_total : 20535 time to create 1 rle with old method : 0.02337503433227539 length of segment : 168 time for calcul the mask position with numpy : 0.0008094310760498047 nb_pixel_total : 30255 time to create 1 rle with old method : 0.03463411331176758 length of segment : 209 time for calcul the mask position with numpy : 0.006907939910888672 nb_pixel_total : 336310 time to create 1 rle with new method : 0.01443934440612793 length of segment : 945 time for calcul the mask position with numpy : 0.0002238750457763672 nb_pixel_total : 7158 time to create 1 rle with old method : 0.008478641510009766 length of segment : 84 time for calcul the mask position with numpy : 0.002050638198852539 nb_pixel_total : 82229 time to create 1 rle with old method : 0.09165620803833008 length of segment : 345 time for calcul the mask position with numpy : 0.010781288146972656 nb_pixel_total : 369049 time to create 1 rle with new method : 0.10353684425354004 length of segment : 1908 time for calcul the mask position with numpy : 0.005859851837158203 nb_pixel_total : 452853 time to create 1 rle with new method : 0.010236501693725586 length of segment : 525 time for calcul the mask position with numpy : 0.0009312629699707031 nb_pixel_total : 34234 time to create 1 rle with old method : 0.03756833076477051 length of segment : 287 time for calcul the mask position with numpy : 0.0009198188781738281 nb_pixel_total : 32005 time to create 1 rle with old method : 0.036919593811035156 length of segment : 226 time for calcul the mask position with numpy : 0.010702848434448242 nb_pixel_total : 425584 time to create 1 rle with new method : 0.027966022491455078 length of segment : 1286 time for calcul the mask position with numpy : 0.001142263412475586 nb_pixel_total : 60347 time to create 1 rle with old method : 0.06876301765441895 length of segment : 294 time for calcul the mask position with numpy : 0.0006103515625 nb_pixel_total : 22041 time to create 1 rle with old method : 0.026080608367919922 length of segment : 170 time for calcul the mask position with numpy : 0.0022869110107421875 nb_pixel_total : 80020 time to create 1 rle with old method : 0.08718752861022949 length of segment : 566 time for calcul the mask position with numpy : 0.006474018096923828 nb_pixel_total : 396033 time to create 1 rle with new method : 0.013785362243652344 length of segment : 988 time for calcul the mask position with numpy : 0.0004665851593017578 nb_pixel_total : 17928 time to create 1 rle with old method : 0.020420551300048828 length of segment : 166 time for calcul the mask position with numpy : 0.0004775524139404297 nb_pixel_total : 16267 time to create 1 rle with old method : 0.01830911636352539 length of segment : 173 time for calcul the mask position with numpy : 0.0010106563568115234 nb_pixel_total : 39394 time to create 1 rle with old method : 0.043480634689331055 length of segment : 278 time for calcul the mask position with numpy : 0.0025169849395751953 nb_pixel_total : 135533 time to create 1 rle with old method : 0.14809250831604004 length of segment : 454 time for calcul the mask position with numpy : 0.002218484878540039 nb_pixel_total : 109373 time to create 1 rle with old method : 0.12543392181396484 length of segment : 419 time spent for convertir_results : 9.011676788330078 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 55 chid ids of type : 3663 Number RLEs to save : 26782 save missing photos in datou_result : time spend for datou_step_exec : 53.98503804206848 time spend to save output : 1.9591643810272217 total time spend for step 1 : 55.9442024230957 step2:crop_condition Tue Feb 18 10:03:26 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 55 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 18 About to insert : list_path_to_insert length 18 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1338303868, 1338303866, 1338303863, 1338303820, 1338303818, 1338303816, 1338303814, 1338303791, 1338303767, 1338303657] Looping around the photos to save general results len do output : 38 /-3678806858Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806871Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806870Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806901Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806905Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806907Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806882Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806892Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806894Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806903Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806887Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806902Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806855Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806857Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806860Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806869Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806865Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806862Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806864Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806872Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806877Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806874Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806884Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806886Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806879Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806890Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806896Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806899Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806900Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806906Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806854Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806859Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806856Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806908Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806863Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806878Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806875Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3678806889Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303868', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303866', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303863', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303820', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303818', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303816', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303814', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303791', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303767', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303657', None, None, None, None, None, '2595746') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 124 time used for this insertion : 0.03710603713989258 save_final save missing photos in datou_result : time spend for datou_step_exec : 36.6126127243042 time spend to save output : 0.03883481025695801 total time spend for step 2 : 36.65144753456116 step3:thcl Tue Feb 18 10:04:03 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.0075075626373291016 time to convert the images to numpy array : 0.027901411056518555 time to import caffe and check if the image exist : 0.009908437728881836 time to convert the images to numpy array : 0.04497170448303223 time to import caffe and check if the image exist : 0.006502866744995117 time to convert the images to numpy array : 0.05391287803649902 time to import caffe and check if the image exist : 0.00962519645690918 time to convert the images to numpy array : 0.06856703758239746 time to import caffe and check if the image exist : 0.011366605758666992 time to convert the images to numpy array : 0.3702850341796875 time to import caffe and check if the image exist : 0.007567644119262695 time to convert the images to numpy array : 0.37933993339538574 time to import caffe and check if the image exist : 0.010283708572387695 time to convert the images to numpy array : 0.7751219272613525 time to import caffe and check if the image exist : 0.01483297348022461 time to convert the images to numpy array : 0.7967042922973633 time to import caffe and check if the image exist : 0.008196353912353516 time to convert the images to numpy array : 0.8104996681213379 time to import caffe and check if the image exist : 0.012338638305664062 time to convert the images to numpy array : 1.24656081199646 total time to convert the images to numpy array : 1.5300278663635254 list photo_ids error: [] list photo_ids correct : [-3678806875, -3678806889, -3678806900, -3678806906, -3678806854, -3678806859, -3678806858, -3678806871, -3678806870, -3678806901, -3678806856, -3678806908, -3678806863, -3678806878, -3678806865, -3678806862, -3678806864, -3678806872, -3678806905, -3678806907, -3678806882, -3678806892, -3678806894, -3678806903, -3678806887, -3678806902, -3678806855, -3678806857, -3678806860, -3678806869, -3678806879, -3678806890, -3678806896, -3678806899, -3678806877, -3678806874, -3678806884, -3678806886] number of photos to traite : 38 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6619 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6400 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.7630059719085693 time used to do the prediction : 0.13891243934631348 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6398 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6398 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.8126397132873535 time used to do the prediction : 0.0849905014038086 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.5735626220703125e-05 save missing photos in datou_result : time spend for datou_step_exec : 10.263625383377075 time spend to save output : 0.00044608116149902344 total time spend for step 3 : 10.264071464538574 step4:merge_mask_thcl_custom Tue Feb 18 10:04:13 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 55 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1338303868, 1338303866, 1338303863, 1338303820, 1338303818, 1338303816, 1338303814, 1338303791, 1338303767, 1338303657] Looping around the photos to save general results len do output : 10 /1338303868Didn't retrieve data .Didn't retrieve data . /1338303866Didn't retrieve data .Didn't retrieve data . /1338303863Didn't retrieve data .Didn't retrieve data . /1338303820Didn't retrieve data .Didn't retrieve data . /1338303818Didn't retrieve data .Didn't retrieve data . /1338303816Didn't retrieve data .Didn't retrieve data . /1338303814Didn't retrieve data .Didn't retrieve data . /1338303791Didn't retrieve data .Didn't retrieve data . /1338303767Didn't retrieve data .Didn't retrieve data . /1338303657Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303868', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303866', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303863', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303820', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303818', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303816', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303814', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303791', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303767', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303657', None, None, None, None, None, '2595746') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013540983200073242 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.05133390426635742 time spend to save output : 0.014103174209594727 total time spend for step 4 : 0.06543707847595215 step5:rle_unique_nms_with_priority Tue Feb 18 10:04:13 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 3 nb_hashtags : 3 time to prepare the origin masks : 4.1992597579956055 time for calcul the mask position with numpy : 0.802264928817749 nb_pixel_total : 8078060 time to create 1 rle with new method : 1.3569531440734863 time for calcul the mask position with numpy : 0.04095149040222168 nb_pixel_total : 23166 time to create 1 rle with old method : 0.025187253952026367 time for calcul the mask position with numpy : 0.02569580078125 nb_pixel_total : 26999 time to create 1 rle with old method : 0.029112577438354492 time for calcul the mask position with numpy : 0.02698040008544922 nb_pixel_total : 166175 time to create 1 rle with new method : 0.7809431552886963 create new chi : 3.164167642593384 time to delete rle : 0.01016998291015625 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 4044 TO DO : save crop sub photo not yet done ! save time : 0.29059815406799316 nb_obj : 2 nb_hashtags : 2 time to prepare the origin masks : 2.48071551322937 time for calcul the mask position with numpy : 0.8432559967041016 nb_pixel_total : 8162660 time to create 1 rle with new method : 0.9006223678588867 time for calcul the mask position with numpy : 0.044084787368774414 nb_pixel_total : 72426 time to create 1 rle with old method : 0.0798490047454834 time for calcul the mask position with numpy : 0.04656338691711426 nb_pixel_total : 59314 time to create 1 rle with old method : 0.06896090507507324 create new chi : 2.0368382930755615 time to delete rle : 0.0004374980926513672 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 3612 TO DO : save crop sub photo not yet done ! save time : 0.30205750465393066 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 8.999287128448486 time for calcul the mask position with numpy : 0.29941272735595703 nb_pixel_total : 7275357 time to create 1 rle with new method : 0.7412054538726807 time for calcul the mask position with numpy : 0.04671001434326172 nb_pixel_total : 55743 time to create 1 rle with old method : 0.06347990036010742 time for calcul the mask position with numpy : 0.04522848129272461 nb_pixel_total : 14544 time to create 1 rle with old method : 0.01627945899963379 time for calcul the mask position with numpy : 0.04370832443237305 nb_pixel_total : 91736 time to create 1 rle with old method : 0.10561013221740723 time for calcul the mask position with numpy : 0.04312586784362793 nb_pixel_total : 13012 time to create 1 rle with old method : 0.01479339599609375 time for calcul the mask position with numpy : 0.07161593437194824 nb_pixel_total : 50186 time to create 1 rle with old method : 0.0728769302368164 time for calcul the mask position with numpy : 0.030962228775024414 nb_pixel_total : 466199 time to create 1 rle with new method : 0.9251928329467773 time for calcul the mask position with numpy : 0.02996683120727539 nb_pixel_total : 68296 time to create 1 rle with old method : 0.08902311325073242 time for calcul the mask position with numpy : 0.02899646759033203 nb_pixel_total : 87966 time to create 1 rle with old method : 0.15381240844726562 time for calcul the mask position with numpy : 0.029140233993530273 nb_pixel_total : 171361 time to create 1 rle with new method : 1.0136339664459229 create new chi : 3.9785687923431396 time to delete rle : 0.0010259151458740234 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 9713 TO DO : save crop sub photo not yet done ! save time : 0.5640501976013184 nb_obj : 8 nb_hashtags : 5 time to prepare the origin masks : 6.266624927520752 time for calcul the mask position with numpy : 1.3026049137115479 nb_pixel_total : 7555209 time to create 1 rle with new method : 0.6592316627502441 time for calcul the mask position with numpy : 0.025677919387817383 nb_pixel_total : 103931 time to create 1 rle with old method : 0.11322188377380371 time for calcul the mask position with numpy : 0.02691030502319336 nb_pixel_total : 26950 time to create 1 rle with old method : 0.029651641845703125 time for calcul the mask position with numpy : 0.02991652488708496 nb_pixel_total : 228393 time to create 1 rle with new method : 0.46397829055786133 time for calcul the mask position with numpy : 0.029178142547607422 nb_pixel_total : 143201 time to create 1 rle with old method : 0.15749692916870117 time for calcul the mask position with numpy : 0.026958227157592773 nb_pixel_total : 88614 time to create 1 rle with old method : 0.09752678871154785 time for calcul the mask position with numpy : 0.026285886764526367 nb_pixel_total : 24214 time to create 1 rle with old method : 0.026526927947998047 time for calcul the mask position with numpy : 0.02745652198791504 nb_pixel_total : 86622 time to create 1 rle with old method : 0.09271502494812012 time for calcul the mask position with numpy : 0.02548837661743164 nb_pixel_total : 37266 time to create 1 rle with old method : 0.04017901420593262 create new chi : 3.281951904296875 time to delete rle : 0.0011272430419921875 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 8736 TO DO : save crop sub photo not yet done ! save time : 0.5079922676086426 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 6.790858268737793 time for calcul the mask position with numpy : 0.4647562503814697 nb_pixel_total : 5986246 time to create 1 rle with new method : 0.8499445915222168 time for calcul the mask position with numpy : 0.02882695198059082 nb_pixel_total : 102329 time to create 1 rle with old method : 0.11945557594299316 time for calcul the mask position with numpy : 0.05165553092956543 nb_pixel_total : 9083 time to create 1 rle with old method : 0.013061285018920898 time for calcul the mask position with numpy : 0.051145076751708984 nb_pixel_total : 85308 time to create 1 rle with old method : 0.09330987930297852 time for calcul the mask position with numpy : 0.06633639335632324 nb_pixel_total : 152542 time to create 1 rle with new method : 0.7134711742401123 time for calcul the mask position with numpy : 0.04471898078918457 nb_pixel_total : 44730 time to create 1 rle with old method : 0.04854917526245117 time for calcul the mask position with numpy : 0.06306791305541992 nb_pixel_total : 1711078 time to create 1 rle with new method : 0.5849552154541016 time for calcul the mask position with numpy : 0.04071760177612305 nb_pixel_total : 78685 time to create 1 rle with old method : 0.08138298988342285 time for calcul the mask position with numpy : 0.0289914608001709 nb_pixel_total : 24127 time to create 1 rle with old method : 0.0261080265045166 time for calcul the mask position with numpy : 0.029620647430419922 nb_pixel_total : 100272 time to create 1 rle with old method : 0.1057441234588623 create new chi : 3.6200904846191406 time to delete rle : 0.001981973648071289 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 15095 TO DO : save crop sub photo not yet done ! save time : 0.8548011779785156 nb_obj : 3 nb_hashtags : 3 time to prepare the origin masks : 2.5086586475372314 time for calcul the mask position with numpy : 1.2986814975738525 nb_pixel_total : 7856355 time to create 1 rle with new method : 0.508314847946167 time for calcul the mask position with numpy : 0.027470111846923828 nb_pixel_total : 20501 time to create 1 rle with old method : 0.02172541618347168 time for calcul the mask position with numpy : 0.028606653213500977 nb_pixel_total : 82076 time to create 1 rle with old method : 0.08730506896972656 time for calcul the mask position with numpy : 0.031555891036987305 nb_pixel_total : 335468 time to create 1 rle with new method : 0.657202959060669 create new chi : 2.739255666732788 time to delete rle : 0.00060272216796875 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 5106 TO DO : save crop sub photo not yet done ! save time : 0.3124690055847168 nb_obj : 2 nb_hashtags : 2 time to prepare the origin masks : 1.9431638717651367 time for calcul the mask position with numpy : 0.5819332599639893 nb_pixel_total : 7473769 time to create 1 rle with new method : 0.7968437671661377 time for calcul the mask position with numpy : 0.04551100730895996 nb_pixel_total : 451964 time to create 1 rle with new method : 0.5465142726898193 time for calcul the mask position with numpy : 0.04734611511230469 nb_pixel_total : 368667 time to create 1 rle with new method : 0.6485152244567871 create new chi : 2.761648178100586 time to delete rle : 0.0009024143218994141 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 7026 TO DO : save crop sub photo not yet done ! save time : 0.43054842948913574 nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 3.1846978664398193 time for calcul the mask position with numpy : 0.5509605407714844 nb_pixel_total : 7808039 time to create 1 rle with new method : 0.7660531997680664 time for calcul the mask position with numpy : 0.026437759399414062 nb_pixel_total : 257 time to create 1 rle with old method : 0.0005090236663818359 time for calcul the mask position with numpy : 0.030307769775390625 nb_pixel_total : 426256 time to create 1 rle with new method : 0.43724846839904785 time for calcul the mask position with numpy : 0.02631068229675293 nb_pixel_total : 59848 time to create 1 rle with old method : 0.06455779075622559 create new chi : 1.9810855388641357 time to delete rle : 0.0005505084991455078 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 5333 TO DO : save crop sub photo not yet done ! save time : 0.3290226459503174 nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 2.9924099445343018 time for calcul the mask position with numpy : 0.566051721572876 nb_pixel_total : 7791385 time to create 1 rle with new method : 0.8123292922973633 time for calcul the mask position with numpy : 0.02895379066467285 nb_pixel_total : 79778 time to create 1 rle with old method : 0.08712577819824219 time for calcul the mask position with numpy : 0.02963542938232422 nb_pixel_total : 21986 time to create 1 rle with old method : 0.023845195770263672 time for calcul the mask position with numpy : 0.03137803077697754 nb_pixel_total : 401251 time to create 1 rle with new method : 0.7218296527862549 create new chi : 2.3908557891845703 time to delete rle : 0.0006382465362548828 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 5430 TO DO : save crop sub photo not yet done ! save time : 0.3261759281158447 nb_obj : 5 nb_hashtags : 4 time to prepare the origin masks : 3.383702039718628 time for calcul the mask position with numpy : 0.4045844078063965 nb_pixel_total : 7976624 time to create 1 rle with new method : 0.5876824855804443 time for calcul the mask position with numpy : 0.028595924377441406 nb_pixel_total : 135123 time to create 1 rle with old method : 0.19761037826538086 time for calcul the mask position with numpy : 0.027736186981201172 nb_pixel_total : 39304 time to create 1 rle with old method : 0.049971580505371094 time for calcul the mask position with numpy : 0.032527923583984375 nb_pixel_total : 109180 time to create 1 rle with old method : 0.15168046951293945 time for calcul the mask position with numpy : 0.029490232467651367 nb_pixel_total : 16270 time to create 1 rle with old method : 0.022004127502441406 time for calcul the mask position with numpy : 0.027536392211914062 nb_pixel_total : 17899 time to create 1 rle with old method : 0.019644975662231445 create new chi : 1.6265292167663574 time to delete rle : 0.0006539821624755859 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 5164 TO DO : save crop sub photo not yet done ! save time : 0.32700443267822266 map_output_result : {1338303868: (0.0, 'Should be the crop_list due to order', 0.0), 1338303866: (0.0, 'Should be the crop_list due to order', 0.0), 1338303863: (0.0, 'Should be the crop_list due to order', 0.0), 1338303820: (0.0, 'Should be the crop_list due to order', 0.0), 1338303818: (0.0, 'Should be the crop_list due to order', 0.0), 1338303816: (0.0, 'Should be the crop_list due to order', 0.0), 1338303814: (0.0, 'Should be the crop_list due to order', 0.0), 1338303791: (0.0, 'Should be the crop_list due to order', 0.0), 1338303767: (0.0, 'Should be the crop_list due to order', 0.0), 1338303657: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1338303868, 1338303866, 1338303863, 1338303820, 1338303818, 1338303816, 1338303814, 1338303791, 1338303767, 1338303657] Looping around the photos to save general results len do output : 10 /1338303868.Didn't retrieve data . /1338303866.Didn't retrieve data . /1338303863.Didn't retrieve data . /1338303820.Didn't retrieve data . /1338303818.Didn't retrieve data . /1338303816.Didn't retrieve data . /1338303814.Didn't retrieve data . /1338303791.Didn't retrieve data . /1338303767.Didn't retrieve data . /1338303657.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303868', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303866', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303863', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303820', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303818', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303816', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303814', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303791', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303767', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303657', None, None, None, None, None, '2595746') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.018117904663085938 save_final save missing photos in datou_result : time spend for datou_step_exec : 75.65493440628052 time spend to save output : 0.01861429214477539 total time spend for step 5 : 75.6735486984253 step6:crop_condition Tue Feb 18 10:05:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 57 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739869532_3017765 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.8537189960479736 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739869537_3017765 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.5420849323272705 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739869542_3017765 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5843768119812012 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739869545_3017765 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7841782569885254 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1338303868, 1338303866, 1338303863, 1338303820, 1338303818, 1338303816, 1338303814, 1338303791, 1338303767, 1338303657] Looping around the photos to save general results len do output : 15 /1338307866Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307867Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307868Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307869Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307870Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307871Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307878Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307879Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307880Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307881Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307882Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307883Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307889Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307890Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338307891Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303868', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303866', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303863', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303820', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303818', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303816', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303814', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303791', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303767', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303657', None, None, None, None, None, '2595746') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 55 time used for this insertion : 0.030886411666870117 save_final save missing photos in datou_result : time spend for datou_step_exec : 16.865837812423706 time spend to save output : 0.03149819374084473 total time spend for step 6 : 16.89733600616455 step7:ventilate_hashtags_in_portfolio Tue Feb 18 10:05:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20631046 get user id for portfolio 20631046 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20631046 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('metal','plastique','Teint_Dans_La_Masse','Carton_brun','environnement','Carton_gris','cartonnette','papier','kraft','mal_croppe','flou','autre_refus')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20631046 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('metal','plastique','Teint_Dans_La_Masse','Carton_brun','environnement','Carton_gris','cartonnette','papier','kraft','mal_croppe','flou','autre_refus')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20631046 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('metal','plastique','Teint_Dans_La_Masse','Carton_brun','environnement','Carton_gris','cartonnette','papier','kraft','mal_croppe','flou','autre_refus')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20631095,20631096,20631097,20631098,20631099,20631100,20631101,20631102,20631103,20631104,20631105,20631106?tags=metal,plastique,Teint_Dans_La_Masse,Carton_brun,environnement,Carton_gris,cartonnette,papier,kraft,mal_croppe,flou,autre_refus Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1338303868, 1338303866, 1338303863, 1338303820, 1338303818, 1338303816, 1338303814, 1338303791, 1338303767, 1338303657] Looping around the photos to save general results len do output : 1 /20631046. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303868', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303866', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303863', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303820', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303818', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303816', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303814', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303791', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303767', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303657', None, None, None, None, None, '2595746') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01455068588256836 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.6139285564422607 time spend to save output : 0.014833688735961914 total time spend for step 7 : 1.6287622451782227 step8:final Tue Feb 18 10:05:47 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1338303868: ('0.04385818526629293',), 1338303866: ('0.04385818526629293',), 1338303863: ('0.04385818526629293',), 1338303820: ('0.04385818526629293',), 1338303818: ('0.04385818526629293',), 1338303816: ('0.04385818526629293',), 1338303814: ('0.04385818526629293',), 1338303791: ('0.04385818526629293',), 1338303767: ('0.04385818526629293',), 1338303657: ('0.04385818526629293',)} new output for save of step final : {1338303868: ('0.04385818526629293',), 1338303866: ('0.04385818526629293',), 1338303863: ('0.04385818526629293',), 1338303820: ('0.04385818526629293',), 1338303818: ('0.04385818526629293',), 1338303816: ('0.04385818526629293',), 1338303814: ('0.04385818526629293',), 1338303791: ('0.04385818526629293',), 1338303767: ('0.04385818526629293',), 1338303657: ('0.04385818526629293',)} [1338303868, 1338303866, 1338303863, 1338303820, 1338303818, 1338303816, 1338303814, 1338303791, 1338303767, 1338303657] Looping around the photos to save general results len do output : 10 /1338303868.Didn't retrieve data . /1338303866.Didn't retrieve data . /1338303863.Didn't retrieve data . /1338303820.Didn't retrieve data . /1338303818.Didn't retrieve data . /1338303816.Didn't retrieve data . /1338303814.Didn't retrieve data . /1338303791.Didn't retrieve data . /1338303767.Didn't retrieve data . /1338303657.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303868', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303866', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303863', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303820', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303818', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303816', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303814', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303791', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303767', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303657', None, None, None, None, None, '2595746') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013950109481811523 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10379457473754883 time spend to save output : 0.014475345611572266 total time spend for step 8 : 0.1182699203491211 step9:velours_tree Tue Feb 18 10:05:47 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.03400135040283203 time spend to save output : 4.673004150390625e-05 total time spend for step 9 : 0.03404808044433594 step10:send_mail_cod Tue Feb 18 10:05:47 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20631046_18-02-2025_10_05_47.pdf 20631095 imagette206310951739869547 20631096 imagette206310961739869547 20631101 imagette206311011739869547 20631103 imagette206311031739869547 20631104 imagette206311041739869547 20631105 imagette206311051739869547 20631106 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette206311061739869547 20631097 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette206310971739869548 20631098 change filename to text .change filename to text .imagette206310981739869548 20631100 change filename to text .imagette206311001739869548 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20631046 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20631095,20631096,20631097,20631098,20631099,20631100,20631101,20631102,20631103,20631104,20631105,20631106?tags=metal,plastique,Teint_Dans_La_Masse,Carton_brun,environnement,Carton_gris,cartonnette,papier,kraft,mal_croppe,flou,autre_refus your option no_mail is active, we will not send the real mail to your client args[1338303868] : ('0.04385818526629293',) no score found for photo 1338303868 apple ('0.04385818526629293',) We are sending mail with results at report@fotonower.com args[1338303866] : ('0.04385818526629293',) no score found for photo 1338303866 apple ('0.04385818526629293',) We are sending mail with results at report@fotonower.com args[1338303863] : ('0.04385818526629293',) no score found for photo 1338303863 apple ('0.04385818526629293',) We are sending mail with results at report@fotonower.com args[1338303820] : ('0.04385818526629293',) no score found for photo 1338303820 apple ('0.04385818526629293',) We are sending mail with results at report@fotonower.com args[1338303818] : ('0.04385818526629293',) no score found for photo 1338303818 apple ('0.04385818526629293',) We are sending mail with results at report@fotonower.com args[1338303816] : ('0.04385818526629293',) no score found for photo 1338303816 apple ('0.04385818526629293',) We are sending mail with results at report@fotonower.com args[1338303814] : ('0.04385818526629293',) no score found for photo 1338303814 apple ('0.04385818526629293',) We are sending mail with results at report@fotonower.com args[1338303791] : ('0.04385818526629293',) no score found for photo 1338303791 apple ('0.04385818526629293',) We are sending mail with results at report@fotonower.com args[1338303767] : ('0.04385818526629293',) no score found for photo 1338303767 apple ('0.04385818526629293',) We are sending mail with results at report@fotonower.com args[1338303657] : ('0.04385818526629293',) no score found for photo 1338303657 apple ('0.04385818526629293',) We are sending mail with results at report@fotonower.com refus_total : 0.04385818526629293 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20631046 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20631046_18-02-2025_10_05_47.pdf results_Auto_P20631046_18-02-2025_10_05_47.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20631046_18-02-2025_10_05_47.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20631046','results_Auto_P20631046_18-02-2025_10_05_47.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20631046_18-02-2025_10_05_47.pdf','pdf','','0.47','0.04385818526629293') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1338303868, 1338303866, 1338303863, 1338303820, 1338303818, 1338303816, 1338303814, 1338303791, 1338303767, 1338303657] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303868', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303866', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303863', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303820', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303818', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303816', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303814', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303791', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303767', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303657', None, None, None, None, None, '2595746') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.015246391296386719 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.3740575313568115 time spend to save output : 0.0155181884765625 total time spend for step 10 : 3.389575719833374 step11:split_time_score Tue Feb 18 10:05:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('09', 22),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 18022025 20631046 Nombre de photos uploadées : 22 / 23040 (0%) 18022025 20631046 Nombre de photos taguées (types de déchets): 0 / 22 (0%) 18022025 20631046 Nombre de photos taguées (volume) : 0 / 22 (0%) elapsed_time : load_data_split_time_score 4.5299530029296875e-06 elapsed_time : order_list_meta_photo_and_scores 1.4543533325195312e-05 ?????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0012371540069580078 elapsed_time : insert_dashboard_record_day_entry 0.026143789291381836 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20631042 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20631043 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20631043 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20631044 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20631045 order by id desc limit 1 Qualite : 0.028295210112685897 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20630385_18-02-2025_10_04_43.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20630385 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20630385 AND mptpi.`type`=3327 To do Qualite : 0.04062884464016455 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20630386_18-02-2025_09_56_54.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20630386 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20630386 AND mptpi.`type`=4207 To do Qualite : 0.04062884464016455 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20630386_18-02-2025_09_56_54.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20630386 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20630386 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20630387 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20630388 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20630389 order by id desc limit 1 Qualite : 0.050898050952396676 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20630390_18-02-2025_10_04_59.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20630390 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20630390 AND mptpi.`type`=4207 To do Qualite : 0.050898050952396676 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20630390_18-02-2025_10_04_59.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20630390 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20630390 AND mptpi.`type`=4207 To do Qualite : 0.05260284935109149 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20630391_18-02-2025_10_01_13.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20630391 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11524 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11533 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11532 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11528 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11528 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11531 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11531 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11578 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11578 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11527 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11526 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11526 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11535 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11527 doesn't seem to be define in the database( WARNING : type of input 3 of step 11526 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11524 doesn't seem to be define in the database( WARNING : type of input 2 of step 11528 doesn't seem to be define in the database( WARNING : output 1 of step 11524 have datatype=2 whereas input 1 of step 11531 have datatype=7 WARNING : type of output 2 of step 11531 doesn't seem to be define in the database( WARNING : type of input 1 of step 11525 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11527 have datatype=10 whereas input 3 of step 11534 have datatype=6 WARNING : type of input 2 of step 11578 doesn't seem to be define in the database( WARNING : output 1 of step 11525 have datatype=7 whereas input 2 of step 11578 have datatype=None WARNING : type of output 3 of step 11578 doesn't seem to be define in the database( WARNING : type of input 1 of step 11527 doesn't seem to be define in the database( WARNING : output 0 of step 11527 have datatype=10 whereas input 0 of step 11584 have datatype=18 WARNING : type of input 5 of step 11534 doesn't seem to be define in the database( WARNING : output 0 of step 11584 have datatype=11 whereas input 5 of step 11534 have datatype=None WARNING : type of output 1 of step 11533 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : type of output 1 of step 11532 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : output 0 of step 11531 have datatype=1 whereas input 0 of step 11525 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20630391 AND mptpi.`type`=4211 To do Qualite : 0.018378179801678333 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20630392_18-02-2025_09_57_22.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20630392 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20630392 AND mptpi.`type`=3327 To do Qualite : 0.04385818526629293 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20631046_18-02-2025_10_05_47.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20631046 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20631046 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20631047 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'18022025': {'nb_upload': 22, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1338303868, 1338303866, 1338303863, 1338303820, 1338303818, 1338303816, 1338303814, 1338303791, 1338303767, 1338303657] Looping around the photos to save general results len do output : 1 /20631046Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303868', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303866', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303863', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303820', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303818', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303816', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303814', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303791', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303767', None, None, None, None, None, '2595746') ('3459', None, None, None, None, None, None, None, '2595746') ('3459', '20631046', '1338303657', None, None, None, None, None, '2595746') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.015317201614379883 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.705091953277588 time spend to save output : 0.015606880187988281 total time spend for step 11 : 7.720698833465576 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 96.15user 96.08system 3:33.94elapsed 89%CPU (0avgtext+0avgdata 3708972maxresident)k 2214016inputs+82520outputs (7989major+8616603minor)pagefaults 0swaps