python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3747315 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 25>10 please execute split_portfolio.py -i 20549745 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2586093'] with mtr_portfolio_ids : ['20549745'] and first list_photo_ids : [] new path : /proc/3747315/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.9467124938964844 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 14 09:36:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-14 09:36:34.263375: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-14 09:36:34.291100: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-14 09:36:34.292516: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f4214000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-14 09:36:34.292538: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-14 09:36:34.295242: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-14 09:36:34.538251: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x42520180 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-14 09:36:34.538302: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-14 09:36:34.539345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-14 09:36:34.539666: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-14 09:36:34.541733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-14 09:36:34.543849: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-14 09:36:34.544184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-14 09:36:34.546376: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-14 09:36:34.547467: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-14 09:36:34.551887: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-14 09:36:34.553363: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-14 09:36:34.553438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-14 09:36:34.554194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-14 09:36:34.554211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-14 09:36:34.554220: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-14 09:36:34.555524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-14 09:36:34.994610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-14 09:36:34.994724: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-14 09:36:34.994753: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-14 09:36:34.994778: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-14 09:36:34.994802: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-14 09:36:34.994827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-14 09:36:34.994850: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-14 09:36:34.994875: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-14 09:36:34.996867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-14 09:36:34.998221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-14 09:36:34.998272: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-14 09:36:34.998291: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-14 09:36:34.998310: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-14 09:36:34.998331: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-14 09:36:34.998351: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-14 09:36:34.998371: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-14 09:36:34.998390: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-14 09:36:34.999974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-14 09:36:35.000007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-14 09:36:35.000018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-14 09:36:35.000028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-14 09:36:35.001646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-14 09:36:47.425266: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-14 09:36:47.613970: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 143.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 142.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 142.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 13 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 141.85000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 140.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 4.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 141.85000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 140.85000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 140.60000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 143.60000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 4.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 139.60000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 4 Detection mask done ! Trying to reset tf kernel 3747920 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1388 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 6677 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0035490989685058594 nb_pixel_total : 134537 time to create 1 rle with old method : 0.14604496955871582 length of segment : 477 time for calcul the mask position with numpy : 0.001085042953491211 nb_pixel_total : 69916 time to create 1 rle with old method : 0.07908868789672852 length of segment : 477 time for calcul the mask position with numpy : 0.0028357505798339844 nb_pixel_total : 180334 time to create 1 rle with new method : 0.00951838493347168 length of segment : 776 time for calcul the mask position with numpy : 0.0006749629974365234 nb_pixel_total : 48753 time to create 1 rle with old method : 0.05515885353088379 length of segment : 203 time for calcul the mask position with numpy : 0.0007059574127197266 nb_pixel_total : 47491 time to create 1 rle with old method : 0.07476687431335449 length of segment : 213 time for calcul the mask position with numpy : 0.00036334991455078125 nb_pixel_total : 17658 time to create 1 rle with old method : 0.020959138870239258 length of segment : 183 time for calcul the mask position with numpy : 0.0007781982421875 nb_pixel_total : 40484 time to create 1 rle with old method : 0.047826528549194336 length of segment : 109 time for calcul the mask position with numpy : 0.0006258487701416016 nb_pixel_total : 48016 time to create 1 rle with old method : 0.053458213806152344 length of segment : 201 time for calcul the mask position with numpy : 0.0012621879577636719 nb_pixel_total : 52627 time to create 1 rle with old method : 0.06365799903869629 length of segment : 748 time for calcul the mask position with numpy : 0.00226593017578125 nb_pixel_total : 129484 time to create 1 rle with old method : 0.1714639663696289 length of segment : 453 time for calcul the mask position with numpy : 0.0010044574737548828 nb_pixel_total : 53349 time to create 1 rle with old method : 0.0661473274230957 length of segment : 415 time for calcul the mask position with numpy : 0.0016562938690185547 nb_pixel_total : 76488 time to create 1 rle with old method : 0.09295463562011719 length of segment : 554 time for calcul the mask position with numpy : 0.001291036605834961 nb_pixel_total : 73222 time to create 1 rle with old method : 0.08646130561828613 length of segment : 613 time for calcul the mask position with numpy : 0.002256155014038086 nb_pixel_total : 128823 time to create 1 rle with old method : 0.1471879482269287 length of segment : 779 time for calcul the mask position with numpy : 0.0010874271392822266 nb_pixel_total : 71249 time to create 1 rle with old method : 0.08187532424926758 length of segment : 295 time for calcul the mask position with numpy : 0.00031876564025878906 nb_pixel_total : 16280 time to create 1 rle with old method : 0.01907181739807129 length of segment : 145 time for calcul the mask position with numpy : 0.00034618377685546875 nb_pixel_total : 16494 time to create 1 rle with old method : 0.01886582374572754 length of segment : 163 time for calcul the mask position with numpy : 0.0008087158203125 nb_pixel_total : 46114 time to create 1 rle with old method : 0.054344892501831055 length of segment : 117 time for calcul the mask position with numpy : 0.0026814937591552734 nb_pixel_total : 134510 time to create 1 rle with old method : 0.1517188549041748 length of segment : 487 time for calcul the mask position with numpy : 0.00017070770263671875 nb_pixel_total : 4089 time to create 1 rle with old method : 0.004867076873779297 length of segment : 221 time for calcul the mask position with numpy : 0.0013604164123535156 nb_pixel_total : 35870 time to create 1 rle with old method : 0.04279041290283203 length of segment : 395 time for calcul the mask position with numpy : 0.0065577030181884766 nb_pixel_total : 254682 time to create 1 rle with new method : 0.01834845542907715 length of segment : 506 time for calcul the mask position with numpy : 0.003643035888671875 nb_pixel_total : 167209 time to create 1 rle with new method : 0.00883340835571289 length of segment : 460 time for calcul the mask position with numpy : 0.02709031105041504 nb_pixel_total : 688627 time to create 1 rle with new method : 0.07267093658447266 length of segment : 3001 time for calcul the mask position with numpy : 0.0012176036834716797 nb_pixel_total : 41545 time to create 1 rle with old method : 0.047968149185180664 length of segment : 275 time for calcul the mask position with numpy : 0.0016162395477294922 nb_pixel_total : 66380 time to create 1 rle with old method : 0.08037900924682617 length of segment : 371 time for calcul the mask position with numpy : 0.0010600090026855469 nb_pixel_total : 16483 time to create 1 rle with old method : 0.020862340927124023 length of segment : 264 time for calcul the mask position with numpy : 0.0008318424224853516 nb_pixel_total : 42740 time to create 1 rle with old method : 0.05397629737854004 length of segment : 320 time for calcul the mask position with numpy : 0.0018253326416015625 nb_pixel_total : 112932 time to create 1 rle with old method : 0.1374378204345703 length of segment : 326 time for calcul the mask position with numpy : 0.008198976516723633 nb_pixel_total : 376710 time to create 1 rle with new method : 0.01448512077331543 length of segment : 1181 time for calcul the mask position with numpy : 0.0014872550964355469 nb_pixel_total : 50914 time to create 1 rle with old method : 0.05968189239501953 length of segment : 343 time for calcul the mask position with numpy : 0.0006647109985351562 nb_pixel_total : 15132 time to create 1 rle with old method : 0.01835346221923828 length of segment : 224 time for calcul the mask position with numpy : 0.0007598400115966797 nb_pixel_total : 31439 time to create 1 rle with old method : 0.03612947463989258 length of segment : 216 time for calcul the mask position with numpy : 0.0018112659454345703 nb_pixel_total : 63728 time to create 1 rle with old method : 0.07582521438598633 length of segment : 581 time for calcul the mask position with numpy : 0.0007746219635009766 nb_pixel_total : 13034 time to create 1 rle with old method : 0.01630425453186035 length of segment : 143 time for calcul the mask position with numpy : 0.006850004196166992 nb_pixel_total : 252051 time to create 1 rle with new method : 0.01806044578552246 length of segment : 1328 time for calcul the mask position with numpy : 0.0020110607147216797 nb_pixel_total : 28464 time to create 1 rle with old method : 0.03323984146118164 length of segment : 488 time for calcul the mask position with numpy : 0.0016055107116699219 nb_pixel_total : 70168 time to create 1 rle with old method : 0.08582472801208496 length of segment : 418 time for calcul the mask position with numpy : 0.0010466575622558594 nb_pixel_total : 39916 time to create 1 rle with old method : 0.05074787139892578 length of segment : 251 time for calcul the mask position with numpy : 0.0002703666687011719 nb_pixel_total : 6893 time to create 1 rle with old method : 0.008738517761230469 length of segment : 97 time for calcul the mask position with numpy : 0.003118753433227539 nb_pixel_total : 160598 time to create 1 rle with new method : 0.005800485610961914 length of segment : 483 time for calcul the mask position with numpy : 0.0005972385406494141 nb_pixel_total : 18960 time to create 1 rle with old method : 0.025292158126831055 length of segment : 204 time for calcul the mask position with numpy : 0.0020012855529785156 nb_pixel_total : 67987 time to create 1 rle with old method : 0.08026647567749023 length of segment : 446 time for calcul the mask position with numpy : 0.0012199878692626953 nb_pixel_total : 57291 time to create 1 rle with old method : 0.06555891036987305 length of segment : 307 time for calcul the mask position with numpy : 0.0021741390228271484 nb_pixel_total : 125398 time to create 1 rle with old method : 0.13775229454040527 length of segment : 430 time for calcul the mask position with numpy : 0.0018613338470458984 nb_pixel_total : 72893 time to create 1 rle with old method : 0.08716011047363281 length of segment : 423 time for calcul the mask position with numpy : 0.003950595855712891 nb_pixel_total : 225986 time to create 1 rle with new method : 0.008556842803955078 length of segment : 749 time for calcul the mask position with numpy : 0.002237081527709961 nb_pixel_total : 83067 time to create 1 rle with old method : 0.09177374839782715 length of segment : 388 time for calcul the mask position with numpy : 0.0013089179992675781 nb_pixel_total : 48067 time to create 1 rle with old method : 0.06716680526733398 length of segment : 583 time for calcul the mask position with numpy : 0.001222848892211914 nb_pixel_total : 54386 time to create 1 rle with old method : 0.061781883239746094 length of segment : 317 time for calcul the mask position with numpy : 0.02478337287902832 nb_pixel_total : 1528729 time to create 1 rle with new method : 0.05889701843261719 length of segment : 2070 time for calcul the mask position with numpy : 0.00047016143798828125 nb_pixel_total : 26028 time to create 1 rle with old method : 0.02986907958984375 length of segment : 175 time for calcul the mask position with numpy : 0.005212306976318359 nb_pixel_total : 338993 time to create 1 rle with new method : 0.011901378631591797 length of segment : 603 time spent for convertir_results : 6.94718599319458 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 53 chid ids of type : 3663 Number RLEs to save : 25995 save missing photos in datou_result : time spend for datou_step_exec : 47.23467779159546 time spend to save output : 1.5054819583892822 total time spend for step 1 : 48.74015974998474 step2:crop_condition Fri Feb 14 09:37:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 53 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 13 About to insert : list_path_to_insert length 13 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1337571552, 1337571549, 1337571522, 1337571515, 1337571488, 1337571485, 1337571450, 1337571442, 1337571441, 1337571436] Looping around the photos to save general results len do output : 34 /-3674815219Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815228Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815239Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815257Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815255Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815258Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815261Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815226Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815225Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815236Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815221Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815231Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815232Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815245Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815247Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815250Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815249Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815254Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815253Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815256Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815262Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815264Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815268Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815223Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815246Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815251Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815270Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815222Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815237Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815233Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815230Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815240Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815266Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674815244Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571552', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571549', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571522', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571515', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571488', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571485', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571450', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571442', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571441', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571436', None, None, None, None, None, '2586093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 112 time used for this insertion : 0.02404952049255371 save_final save missing photos in datou_result : time spend for datou_step_exec : 20.554410457611084 time spend to save output : 0.025772809982299805 total time spend for step 2 : 20.580183267593384 step3:thcl Fri Feb 14 09:37:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.005833864212036133 time to convert the images to numpy array : 2.384185791015625e-06 time to import caffe and check if the image exist : 0.005304813385009766 time to convert the images to numpy array : 0.03871965408325195 time to import caffe and check if the image exist : 0.007582664489746094 time to convert the images to numpy array : 0.03807473182678223 time to import caffe and check if the image exist : 0.01188349723815918 time to convert the images to numpy array : 0.03962278366088867 time to import caffe and check if the image exist : 0.00712132453918457 time to convert the images to numpy array : 0.04557466506958008 time to import caffe and check if the image exist : 0.004754304885864258 time to convert the images to numpy array : 0.05301237106323242 time to import caffe and check if the image exist : 0.01217794418334961 time to convert the images to numpy array : 0.05255579948425293 time to import caffe and check if the image exist : 0.010647058486938477 time to convert the images to numpy array : 0.05717825889587402 time to import caffe and check if the image exist : 0.009694337844848633 time to convert the images to numpy array : 0.06380701065063477 time to import caffe and check if the image exist : 0.006226778030395508 time to convert the images to numpy array : 0.08371615409851074 total time to convert the images to numpy array : 0.3016078472137451 list photo_ids error: [] list photo_ids correct : [-3674815246, -3674815251, -3674815270, -3674815222, -3674815266, -3674815244, -3674815255, -3674815258, -3674815261, -3674815226, -3674815262, -3674815264, -3674815268, -3674815223, -3674815219, -3674815228, -3674815239, -3674815257, -3674815232, -3674815245, -3674815247, -3674815250, -3674815225, -3674815236, -3674815221, -3674815231, -3674815249, -3674815254, -3674815253, -3674815256, -3674815237, -3674815233, -3674815230, -3674815240] number of photos to traite : 34 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3125 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2906 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.46480393409729004 time used to do the prediction : 0.12842845916748047 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2904 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2904 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.3553338050842285 time used to do the prediction : 0.08627867698669434 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 8.58306884765625e-06 save missing photos in datou_result : time spend for datou_step_exec : 7.880855083465576 time spend to save output : 0.0003237724304199219 total time spend for step 3 : 7.881178855895996 step4:merge_mask_thcl_custom Fri Feb 14 09:37:48 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 53 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1337571552, 1337571549, 1337571522, 1337571515, 1337571488, 1337571485, 1337571450, 1337571442, 1337571441, 1337571436] Looping around the photos to save general results len do output : 10 /1337571552Didn't retrieve data .Didn't retrieve data . /1337571549Didn't retrieve data .Didn't retrieve data . /1337571522Didn't retrieve data .Didn't retrieve data . /1337571515Didn't retrieve data .Didn't retrieve data . /1337571488Didn't retrieve data .Didn't retrieve data . /1337571485Didn't retrieve data .Didn't retrieve data . /1337571450Didn't retrieve data .Didn't retrieve data . /1337571442Didn't retrieve data .Didn't retrieve data . /1337571441Didn't retrieve data .Didn't retrieve data . /1337571436Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571552', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571549', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571522', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571515', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571488', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571485', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571450', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571442', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571441', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571436', None, None, None, None, None, '2586093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013691902160644531 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.0517885684967041 time spend to save output : 0.01422262191772461 total time spend for step 4 : 0.06601119041442871 step5:rle_unique_nms_with_priority Fri Feb 14 09:37:48 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 7 nb_hashtags : 2 time to prepare the origin masks : 3.1939570903778076 time for calcul the mask position with numpy : 0.3023855686187744 nb_pixel_total : 7773086 time to create 1 rle with new method : 0.41003942489624023 time for calcul the mask position with numpy : 0.023468017578125 nb_pixel_total : 1427 time to create 1 rle with old method : 0.0019075870513916016 time for calcul the mask position with numpy : 0.02486729621887207 nb_pixel_total : 47972 time to create 1 rle with old method : 0.053563833236694336 time for calcul the mask position with numpy : 0.025740861892700195 nb_pixel_total : 180224 time to create 1 rle with new method : 0.3388400077819824 time for calcul the mask position with numpy : 0.024009227752685547 nb_pixel_total : 40287 time to create 1 rle with old method : 0.04443717002868652 time for calcul the mask position with numpy : 0.02472710609436035 nb_pixel_total : 134107 time to create 1 rle with old method : 0.14764022827148438 time for calcul the mask position with numpy : 0.02454972267150879 nb_pixel_total : 47373 time to create 1 rle with old method : 0.05600404739379883 time for calcul the mask position with numpy : 0.026145219802856445 nb_pixel_total : 69924 time to create 1 rle with old method : 0.07916975021362305 create new chi : 1.6653215885162354 time to delete rle : 0.009391069412231445 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6843 TO DO : save crop sub photo not yet done ! save time : 0.3996446132659912 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 4.923060655593872 time for calcul the mask position with numpy : 0.32949066162109375 nb_pixel_total : 7566868 time to create 1 rle with new method : 0.31220221519470215 time for calcul the mask position with numpy : 0.024344205856323242 nb_pixel_total : 133926 time to create 1 rle with old method : 0.14673924446105957 time for calcul the mask position with numpy : 0.024251222610473633 nb_pixel_total : 71165 time to create 1 rle with old method : 0.08740901947021484 time for calcul the mask position with numpy : 0.024682283401489258 nb_pixel_total : 16249 time to create 1 rle with old method : 0.01870250701904297 time for calcul the mask position with numpy : 0.026587963104248047 nb_pixel_total : 76617 time to create 1 rle with old method : 0.08527684211730957 time for calcul the mask position with numpy : 0.02828836441040039 nb_pixel_total : 129227 time to create 1 rle with old method : 0.14653611183166504 time for calcul the mask position with numpy : 0.025249481201171875 nb_pixel_total : 45849 time to create 1 rle with old method : 0.05188393592834473 time for calcul the mask position with numpy : 0.024649620056152344 nb_pixel_total : 128112 time to create 1 rle with old method : 0.1422438621520996 time for calcul the mask position with numpy : 0.02443408966064453 nb_pixel_total : 53282 time to create 1 rle with old method : 0.059235572814941406 time for calcul the mask position with numpy : 0.024592161178588867 nb_pixel_total : 73105 time to create 1 rle with old method : 0.08032870292663574 create new chi : 1.7352824211120605 time to delete rle : 0.0008308887481689453 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 9818 TO DO : save crop sub photo not yet done ! save time : 0.5816900730133057 nb_obj : 7 nb_hashtags : 4 time to prepare the origin masks : 5.934709787368774 time for calcul the mask position with numpy : 0.3211688995361328 nb_pixel_total : 7221643 time to create 1 rle with new method : 0.4504854679107666 time for calcul the mask position with numpy : 0.02871847152709961 nb_pixel_total : 20938 time to create 1 rle with old method : 0.03764462471008301 time for calcul the mask position with numpy : 0.03315138816833496 nb_pixel_total : 544056 time to create 1 rle with new method : 0.41765260696411133 time for calcul the mask position with numpy : 0.02660655975341797 nb_pixel_total : 167130 time to create 1 rle with new method : 0.3742671012878418 time for calcul the mask position with numpy : 0.026231050491333008 nb_pixel_total : 66098 time to create 1 rle with old method : 0.07862448692321777 time for calcul the mask position with numpy : 0.026625394821166992 nb_pixel_total : 254059 time to create 1 rle with new method : 0.5078964233398438 time for calcul the mask position with numpy : 0.026908397674560547 nb_pixel_total : 4075 time to create 1 rle with old method : 0.00475764274597168 time for calcul the mask position with numpy : 0.027068614959716797 nb_pixel_total : 16401 time to create 1 rle with old method : 0.019194364547729492 create new chi : 2.5377421379089355 time to delete rle : 0.0009951591491699219 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 11915 TO DO : save crop sub photo not yet done ! save time : 0.673774003982544 nb_obj : 5 nb_hashtags : 4 time to prepare the origin masks : 3.374655246734619 time for calcul the mask position with numpy : 0.46520256996154785 nb_pixel_total : 7696767 time to create 1 rle with new method : 0.40096306800842285 time for calcul the mask position with numpy : 0.03190135955810547 nb_pixel_total : 50921 time to create 1 rle with old method : 0.05779075622558594 time for calcul the mask position with numpy : 0.027530908584594727 nb_pixel_total : 42706 time to create 1 rle with old method : 0.05379629135131836 time for calcul the mask position with numpy : 0.027413368225097656 nb_pixel_total : 112443 time to create 1 rle with old method : 0.1331489086151123 time for calcul the mask position with numpy : 0.03215789794921875 nb_pixel_total : 376466 time to create 1 rle with new method : 0.4585545063018799 time for calcul the mask position with numpy : 0.02745652198791504 nb_pixel_total : 15097 time to create 1 rle with old method : 0.017367124557495117 create new chi : 1.8180887699127197 time to delete rle : 0.001280069351196289 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6992 TO DO : save crop sub photo not yet done ! save time : 0.4151484966278076 nb_obj : 4 nb_hashtags : 4 time to prepare the origin masks : 3.358036994934082 time for calcul the mask position with numpy : 0.4070758819580078 nb_pixel_total : 7928609 time to create 1 rle with new method : 0.4298722743988037 time for calcul the mask position with numpy : 0.03359103202819824 nb_pixel_total : 257933 time to create 1 rle with new method : 0.5638680458068848 time for calcul the mask position with numpy : 0.03771471977233887 nb_pixel_total : 12826 time to create 1 rle with old method : 0.015219926834106445 time for calcul the mask position with numpy : 0.027187108993530273 nb_pixel_total : 31428 time to create 1 rle with old method : 0.03760552406311035 time for calcul the mask position with numpy : 0.027004480361938477 nb_pixel_total : 63604 time to create 1 rle with old method : 0.07496166229248047 create new chi : 1.7307205200195312 time to delete rle : 0.0009064674377441406 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 6184 TO DO : save crop sub photo not yet done ! save time : 0.3711836338043213 nb_obj : 4 nb_hashtags : 4 time to prepare the origin masks : 3.8109612464904785 time for calcul the mask position with numpy : 0.38068318367004395 nb_pixel_total : 8149320 time to create 1 rle with new method : 0.43753576278686523 time for calcul the mask position with numpy : 0.04419660568237305 nb_pixel_total : 69919 time to create 1 rle with old method : 0.08061051368713379 time for calcul the mask position with numpy : 0.04223895072937012 nb_pixel_total : 28454 time to create 1 rle with old method : 0.032713890075683594 time for calcul the mask position with numpy : 0.04250764846801758 nb_pixel_total : 39831 time to create 1 rle with old method : 0.04782223701477051 time for calcul the mask position with numpy : 0.02615070343017578 nb_pixel_total : 6876 time to create 1 rle with old method : 0.008001089096069336 create new chi : 1.200402021408081 time to delete rle : 0.00047779083251953125 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 4626 TO DO : save crop sub photo not yet done ! save time : 0.29031968116760254 nb_obj : 4 nb_hashtags : 3 time to prepare the origin masks : 2.803097724914551 time for calcul the mask position with numpy : 0.3109431266784668 nb_pixel_total : 8078441 time to create 1 rle with new method : 0.3706204891204834 time for calcul the mask position with numpy : 0.03381609916687012 nb_pixel_total : 67804 time to create 1 rle with old method : 0.0811314582824707 time for calcul the mask position with numpy : 0.026997804641723633 nb_pixel_total : 57140 time to create 1 rle with old method : 0.07078146934509277 time for calcul the mask position with numpy : 0.028714418411254883 nb_pixel_total : 18889 time to create 1 rle with old method : 0.02176952362060547 time for calcul the mask position with numpy : 0.044497013092041016 nb_pixel_total : 72126 time to create 1 rle with old method : 0.08436870574951172 create new chi : 1.1397905349731445 time to delete rle : 0.0005295276641845703 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 4894 TO DO : save crop sub photo not yet done ! save time : 0.3166968822479248 nb_obj : 2 nb_hashtags : 2 time to prepare the origin masks : 1.811964511871338 time for calcul the mask position with numpy : 0.4456315040588379 nb_pixel_total : 7985691 time to create 1 rle with new method : 0.6064395904541016 time for calcul the mask position with numpy : 0.027345895767211914 nb_pixel_total : 82872 time to create 1 rle with old method : 0.09430551528930664 time for calcul the mask position with numpy : 0.026501893997192383 nb_pixel_total : 225837 time to create 1 rle with new method : 0.7269902229309082 create new chi : 1.99983549118042 time to delete rle : 0.0007784366607666016 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 4450 TO DO : save crop sub photo not yet done ! save time : 0.2865328788757324 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 1.3584399223327637 time for calcul the mask position with numpy : 0.2794966697692871 nb_pixel_total : 8192144 time to create 1 rle with new method : 0.7356634140014648 time for calcul the mask position with numpy : 0.024410724639892578 nb_pixel_total : 54225 time to create 1 rle with old method : 0.061258554458618164 time for calcul the mask position with numpy : 0.02464914321899414 nb_pixel_total : 48031 time to create 1 rle with old method : 0.05596351623535156 create new chi : 1.2223842144012451 time to delete rle : 0.00052642822265625 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 3954 TO DO : save crop sub photo not yet done ! save time : 0.2463998794555664 nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 2.9659252166748047 time for calcul the mask position with numpy : 0.4625697135925293 nb_pixel_total : 6554145 time to create 1 rle with new method : 0.6598351001739502 time for calcul the mask position with numpy : 0.026454448699951172 nb_pixel_total : 25931 time to create 1 rle with old method : 0.03272104263305664 time for calcul the mask position with numpy : 0.0269925594329834 nb_pixel_total : 166154 time to create 1 rle with new method : 0.46134424209594727 time for calcul the mask position with numpy : 0.035681724548339844 nb_pixel_total : 1548170 time to create 1 rle with new method : 0.5279765129089355 create new chi : 2.3325111865997314 time to delete rle : 0.0007543563842773438 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 7082 TO DO : save crop sub photo not yet done ! save time : 0.41295409202575684 map_output_result : {1337571552: (0.0, 'Should be the crop_list due to order', 0.0), 1337571549: (0.0, 'Should be the crop_list due to order', 0.0), 1337571522: (0.0, 'Should be the crop_list due to order', 0.0), 1337571515: (0.0, 'Should be the crop_list due to order', 0.0), 1337571488: (0.0, 'Should be the crop_list due to order', 0.0), 1337571485: (0.0, 'Should be the crop_list due to order', 0.0), 1337571450: (0.0, 'Should be the crop_list due to order', 0.0), 1337571442: (0.0, 'Should be the crop_list due to order', 0.0), 1337571441: (0.0, 'Should be the crop_list due to order', 0.0), 1337571436: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1337571552, 1337571549, 1337571522, 1337571515, 1337571488, 1337571485, 1337571450, 1337571442, 1337571441, 1337571436] Looping around the photos to save general results len do output : 10 /1337571552.Didn't retrieve data . /1337571549.Didn't retrieve data . /1337571522.Didn't retrieve data . /1337571515.Didn't retrieve data . /1337571488.Didn't retrieve data . /1337571485.Didn't retrieve data . /1337571450.Didn't retrieve data . /1337571442.Didn't retrieve data . /1337571441.Didn't retrieve data . /1337571436.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571552', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571549', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571522', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571515', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571488', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571485', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571450', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571442', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571441', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571436', None, None, None, None, None, '2586093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01273036003112793 save_final save missing photos in datou_result : time spend for datou_step_exec : 56.09258794784546 time spend to save output : 0.013228893280029297 total time spend for step 5 : 56.10581684112549 step6:crop_condition Fri Feb 14 09:38:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 57 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739522329_3747315 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.6964986324310303 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739522332_3747315 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.1239068508148193 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739522335_3747315 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.570960283279419 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739522337_3747315 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.563391923904419 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 map_result returned by crop_photo_return_map_crop : length : 0 About to insert : list_path_to_insert length 0 new photo from crops ! About to upload 0 photos WARNING : list_path_to_insert is empty, cannot upload ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1337571552, 1337571549, 1337571522, 1337571515, 1337571488, 1337571485, 1337571450, 1337571442, 1337571441, 1337571436] Looping around the photos to save general results len do output : 13 /1337580953Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337580954Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337580955Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337580956Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337580957Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337580958Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337580960Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337580964Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337580965Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337580966Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337580967Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337580970Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337580972Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571552', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571549', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571522', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571515', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571488', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571485', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571450', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571442', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571441', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571436', None, None, None, None, None, '2586093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 49 time used for this insertion : 0.02411961555480957 save_final save missing photos in datou_result : time spend for datou_step_exec : 14.172613382339478 time spend to save output : 0.024993419647216797 total time spend for step 6 : 14.197606801986694 step7:ventilate_hashtags_in_portfolio Fri Feb 14 09:38:58 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20549745 get user id for portfolio 20549745 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20549745 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('metal','papier','mal_croppe','Carton_gris','autre_refus','Carton_brun','Teint_Dans_La_Masse','kraft','cartonnette','flou','plastique','environnement')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20549745 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('metal','papier','mal_croppe','Carton_gris','autre_refus','Carton_brun','Teint_Dans_La_Masse','kraft','cartonnette','flou','plastique','environnement')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20549803,20549804,20549805,20549806,20549807,20549808,20549809,20549810,20549811,20549812,20549813,20549814?tags=Carton_gris,Carton_brun,flou,kraft,metal,environnement,autre_refus,Teint_Dans_La_Masse,plastique,mal_croppe,cartonnette,papier Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1337571552, 1337571549, 1337571522, 1337571515, 1337571488, 1337571485, 1337571450, 1337571442, 1337571441, 1337571436] Looping around the photos to save general results len do output : 1 /20549745. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571552', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571549', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571522', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571515', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571488', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571485', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571450', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571442', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571441', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571436', None, None, None, None, None, '2586093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013714075088500977 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.3465275764465332 time spend to save output : 0.013963699340820312 total time spend for step 7 : 1.3604912757873535 step8:final Fri Feb 14 09:39:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1337571552: ('0.035663127432718475',), 1337571549: ('0.035663127432718475',), 1337571522: ('0.035663127432718475',), 1337571515: ('0.035663127432718475',), 1337571488: ('0.035663127432718475',), 1337571485: ('0.035663127432718475',), 1337571450: ('0.035663127432718475',), 1337571442: ('0.035663127432718475',), 1337571441: ('0.035663127432718475',), 1337571436: ('0.035663127432718475',)} new output for save of step final : {1337571552: ('0.035663127432718475',), 1337571549: ('0.035663127432718475',), 1337571522: ('0.035663127432718475',), 1337571515: ('0.035663127432718475',), 1337571488: ('0.035663127432718475',), 1337571485: ('0.035663127432718475',), 1337571450: ('0.035663127432718475',), 1337571442: ('0.035663127432718475',), 1337571441: ('0.035663127432718475',), 1337571436: ('0.035663127432718475',)} [1337571552, 1337571549, 1337571522, 1337571515, 1337571488, 1337571485, 1337571450, 1337571442, 1337571441, 1337571436] Looping around the photos to save general results len do output : 10 /1337571552.Didn't retrieve data . /1337571549.Didn't retrieve data . /1337571522.Didn't retrieve data . /1337571515.Didn't retrieve data . /1337571488.Didn't retrieve data . /1337571485.Didn't retrieve data . /1337571450.Didn't retrieve data . /1337571442.Didn't retrieve data . /1337571441.Didn't retrieve data . /1337571436.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571552', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571549', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571522', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571515', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571488', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571485', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571450', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571442', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571441', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571436', None, None, None, None, None, '2586093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013428449630737305 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.09983658790588379 time spend to save output : 0.014016151428222656 total time spend for step 8 : 0.11385273933410645 step9:velours_tree Fri Feb 14 09:39:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.030946731567382812 time spend to save output : 4.649162292480469e-05 total time spend for step 9 : 0.030993223190307617 step10:send_mail_cod Fri Feb 14 09:39:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20549745_14-02-2025_09_39_00.pdf 20549805 imagette205498051739522340 20549806 imagette205498061739522340 20549807 imagette205498071739522340 20549809 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette205498091739522340 20549811 imagette205498111739522341 20549812 imagette205498121739522341 20549813 imagette205498131739522341 20549803 change filename to text .change filename to text .change filename to text .imagette205498031739522341 20549804 change filename to text .change filename to text .imagette205498041739522341 20549810 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette205498101739522341 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20549745 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20549803,20549804,20549805,20549806,20549807,20549808,20549809,20549810,20549811,20549812,20549813,20549814?tags=Carton_gris,Carton_brun,flou,kraft,metal,environnement,autre_refus,Teint_Dans_La_Masse,plastique,mal_croppe,cartonnette,papier your option no_mail is active, we will not send the real mail to your client args[1337571552] : ('0.035663127432718475',) no score found for photo 1337571552 apple ('0.035663127432718475',) We are sending mail with results at report@fotonower.com args[1337571549] : ('0.035663127432718475',) no score found for photo 1337571549 apple ('0.035663127432718475',) We are sending mail with results at report@fotonower.com args[1337571522] : ('0.035663127432718475',) no score found for photo 1337571522 apple ('0.035663127432718475',) We are sending mail with results at report@fotonower.com args[1337571515] : ('0.035663127432718475',) no score found for photo 1337571515 apple ('0.035663127432718475',) We are sending mail with results at report@fotonower.com args[1337571488] : ('0.035663127432718475',) no score found for photo 1337571488 apple ('0.035663127432718475',) We are sending mail with results at report@fotonower.com args[1337571485] : ('0.035663127432718475',) no score found for photo 1337571485 apple ('0.035663127432718475',) We are sending mail with results at report@fotonower.com args[1337571450] : ('0.035663127432718475',) no score found for photo 1337571450 apple ('0.035663127432718475',) We are sending mail with results at report@fotonower.com args[1337571442] : ('0.035663127432718475',) no score found for photo 1337571442 apple ('0.035663127432718475',) We are sending mail with results at report@fotonower.com args[1337571441] : ('0.035663127432718475',) no score found for photo 1337571441 apple ('0.035663127432718475',) We are sending mail with results at report@fotonower.com args[1337571436] : ('0.035663127432718475',) no score found for photo 1337571436 apple ('0.035663127432718475',) We are sending mail with results at report@fotonower.com refus_total : 0.035663127432718475 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20549745 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20549745_14-02-2025_09_39_00.pdf results_Auto_P20549745_14-02-2025_09_39_00.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20549745_14-02-2025_09_39_00.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20549745','results_Auto_P20549745_14-02-2025_09_39_00.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20549745_14-02-2025_09_39_00.pdf','pdf','','0.56','0.035663127432718475') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1337571552, 1337571549, 1337571522, 1337571515, 1337571488, 1337571485, 1337571450, 1337571442, 1337571441, 1337571436] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571552', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571549', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571522', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571515', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571488', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571485', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571450', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571442', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571441', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571436', None, None, None, None, None, '2586093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.013317584991455078 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.779423475265503 time spend to save output : 0.013690710067749023 total time spend for step 10 : 3.793114185333252 step11:split_time_score Fri Feb 14 09:39:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('08', 25),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 14022025 20549745 Nombre de photos uploadées : 25 / 23040 (0%) 14022025 20549745 Nombre de photos taguées (types de déchets): 0 / 25 (0%) 14022025 20549745 Nombre de photos taguées (volume) : 0 / 25 (0%) elapsed_time : load_data_split_time_score 6.9141387939453125e-06 elapsed_time : order_list_meta_photo_and_scores 7.152557373046875e-06 ????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0012886524200439453 elapsed_time : insert_dashboard_record_day_entry 0.023416519165039062 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.02707671464965672 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20544936_14-02-2025_07_51_52.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20544936 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20544936 AND mptpi.`type`=3726 To do Qualite : 0.0506002903548881 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20544937_14-02-2025_08_35_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20544937 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11524 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11533 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11532 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11528 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11528 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11531 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11531 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11578 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11578 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11527 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11526 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11526 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11535 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11527 doesn't seem to be define in the database( WARNING : type of input 3 of step 11526 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11524 doesn't seem to be define in the database( WARNING : type of input 2 of step 11528 doesn't seem to be define in the database( WARNING : output 1 of step 11524 have datatype=2 whereas input 1 of step 11531 have datatype=7 WARNING : type of output 2 of step 11531 doesn't seem to be define in the database( WARNING : type of input 1 of step 11525 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11527 have datatype=10 whereas input 3 of step 11534 have datatype=6 WARNING : type of input 2 of step 11578 doesn't seem to be define in the database( WARNING : output 1 of step 11525 have datatype=7 whereas input 2 of step 11578 have datatype=None WARNING : type of output 3 of step 11578 doesn't seem to be define in the database( WARNING : type of input 1 of step 11527 doesn't seem to be define in the database( WARNING : output 0 of step 11527 have datatype=10 whereas input 0 of step 11584 have datatype=18 WARNING : type of input 5 of step 11534 doesn't seem to be define in the database( WARNING : output 0 of step 11584 have datatype=11 whereas input 5 of step 11534 have datatype=None WARNING : type of output 1 of step 11533 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : type of output 1 of step 11532 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : output 0 of step 11531 have datatype=1 whereas input 0 of step 11525 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20544937 AND mptpi.`type`=4211 To do Qualite : 0.12087122216415642 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20544938_14-02-2025_07_49_12.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20544938 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20544938 AND mptpi.`type`=3726 To do Qualite : 4.5173863185476346e-05 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20544939_14-02-2025_09_12_57.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20544939 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20544939 AND mptpi.`type`=3327 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20544940 order by id desc limit 1 Qualite : 0.346444078437983 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20544941_14-02-2025_08_48_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20544941 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20544941 AND mptpi.`type`=4200 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20547838 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20547838 order by id desc limit 1 Qualite : 0.059747700372597676 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20547839_14-02-2025_09_35_40.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20547839 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11524 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11533 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11532 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11528 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11528 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11531 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11531 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11578 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11578 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11527 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11526 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11526 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11535 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11527 doesn't seem to be define in the database( WARNING : type of input 3 of step 11526 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11524 doesn't seem to be define in the database( WARNING : type of input 2 of step 11528 doesn't seem to be define in the database( WARNING : output 1 of step 11524 have datatype=2 whereas input 1 of step 11531 have datatype=7 WARNING : type of output 2 of step 11531 doesn't seem to be define in the database( WARNING : type of input 1 of step 11525 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11527 have datatype=10 whereas input 3 of step 11534 have datatype=6 WARNING : type of input 2 of step 11578 doesn't seem to be define in the database( WARNING : output 1 of step 11525 have datatype=7 whereas input 2 of step 11578 have datatype=None WARNING : type of output 3 of step 11578 doesn't seem to be define in the database( WARNING : type of input 1 of step 11527 doesn't seem to be define in the database( WARNING : output 0 of step 11527 have datatype=10 whereas input 0 of step 11584 have datatype=18 WARNING : type of input 5 of step 11534 doesn't seem to be define in the database( WARNING : output 0 of step 11584 have datatype=11 whereas input 5 of step 11534 have datatype=None WARNING : type of output 1 of step 11533 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : type of output 1 of step 11532 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : output 0 of step 11531 have datatype=1 whereas input 0 of step 11525 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20547839 AND mptpi.`type`=4211 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20547840 order by id desc limit 1 Qualite : 0.002868135325500819 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20547841_14-02-2025_09_05_31.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20547841 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20547841 AND mptpi.`type`=3327 To do Qualite : 0.0635499957978284 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20547842_14-02-2025_08_53_38.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20547842 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20547842 AND mptpi.`type`=4207 To do Qualite : 0.0635499957978284 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20547842_14-02-2025_08_53_38.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20547842 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20547842 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20549743 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20549744 order by id desc limit 1 Qualite : 0.04899331568735233 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20549745_14-02-2025_09_39_00.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20549745 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20549745 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20549746 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20549747 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20549747 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20549748 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'14022025': {'nb_upload': 25, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1337571552, 1337571549, 1337571522, 1337571515, 1337571488, 1337571485, 1337571450, 1337571442, 1337571441, 1337571436] Looping around the photos to save general results len do output : 1 /20549745Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571552', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571549', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571522', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571515', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571488', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571485', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571450', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571442', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571441', None, None, None, None, None, '2586093') ('3459', None, None, None, None, None, None, None, '2586093') ('3459', None, '1337571436', None, None, None, None, None, '2586093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014520645141601562 save_final save missing photos in datou_result : time spend for datou_step_exec : 8.600389957427979 time spend to save output : 0.014750480651855469 total time spend for step 11 : 8.615140438079834 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 80.52user 67.01system 2:47.13elapsed 88%CPU (0avgtext+0avgdata 3642556maxresident)k 475720inputs+72048outputs (218major+7388699minor)pagefaults 0swaps