python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2350145 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 15>10 please execute split_portfolio.py -i 20901284 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2621501'] with mtr_portfolio_ids : ['20901284'] and first list_photo_ids : [] new path : /proc/2350145/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.2048282623291016 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 28 01:44:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-28 01:44:34.721240: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-28 01:44:34.747099: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-28 01:44:34.748631: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fb1d8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-28 01:44:34.748652: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-28 01:44:34.751939: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-28 01:44:35.012287: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x116b9a60 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-28 01:44:35.012371: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-28 01:44:35.013468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 01:44:35.014013: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 01:44:35.021559: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 01:44:35.024273: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 01:44:35.024693: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 01:44:35.027730: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 01:44:35.029210: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 01:44:35.035479: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 01:44:35.037005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 01:44:35.037102: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 01:44:35.037840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-28 01:44:35.037860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-28 01:44:35.037872: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-28 01:44:35.039214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4631 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-28 01:44:35.369472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 01:44:35.369585: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 01:44:35.369607: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 01:44:35.369626: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 01:44:35.369644: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 01:44:35.369662: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 01:44:35.369679: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 01:44:35.369697: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 01:44:35.370887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 01:44:35.372307: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 01:44:35.372365: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 01:44:35.372385: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 01:44:35.372402: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 01:44:35.372419: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 01:44:35.372436: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 01:44:35.372452: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 01:44:35.372469: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 01:44:35.373621: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 01:44:35.373671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-28 01:44:35.373682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-28 01:44:35.373692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-28 01:44:35.374947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4631 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-28 01:44:45.826010: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 01:44:46.050403: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 01:44:47.457795: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.09576 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 24 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 7 Detection mask done ! Trying to reset tf kernel 2350799 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 423 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5304 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.002152681350708008 nb_pixel_total : 77370 time to create 1 rle with old method : 0.09371542930603027 length of segment : 335 time for calcul the mask position with numpy : 0.0010530948638916016 nb_pixel_total : 56981 time to create 1 rle with old method : 0.06975269317626953 length of segment : 505 time for calcul the mask position with numpy : 0.2777235507965088 nb_pixel_total : 1331194 time to create 1 rle with new method : 0.11233091354370117 length of segment : 2584 time for calcul the mask position with numpy : 0.0006482601165771484 nb_pixel_total : 27504 time to create 1 rle with old method : 0.03382396697998047 length of segment : 362 time for calcul the mask position with numpy : 0.009440422058105469 nb_pixel_total : 668434 time to create 1 rle with new method : 0.04546403884887695 length of segment : 1285 time for calcul the mask position with numpy : 0.0006389617919921875 nb_pixel_total : 57659 time to create 1 rle with old method : 0.06734347343444824 length of segment : 148 time for calcul the mask position with numpy : 0.0003077983856201172 nb_pixel_total : 11252 time to create 1 rle with old method : 0.013368368148803711 length of segment : 270 time for calcul the mask position with numpy : 0.00014328956604003906 nb_pixel_total : 2331 time to create 1 rle with old method : 0.002920866012573242 length of segment : 79 time for calcul the mask position with numpy : 0.008744478225708008 nb_pixel_total : 369268 time to create 1 rle with new method : 0.04989337921142578 length of segment : 1462 time for calcul the mask position with numpy : 0.0004515647888183594 nb_pixel_total : 19981 time to create 1 rle with old method : 0.023993253707885742 length of segment : 203 time for calcul the mask position with numpy : 0.00018835067749023438 nb_pixel_total : 6163 time to create 1 rle with old method : 0.007136821746826172 length of segment : 195 time for calcul the mask position with numpy : 0.00043010711669921875 nb_pixel_total : 27535 time to create 1 rle with old method : 0.033045291900634766 length of segment : 208 time for calcul the mask position with numpy : 0.0013577938079833984 nb_pixel_total : 59490 time to create 1 rle with old method : 0.09813475608825684 length of segment : 383 time for calcul the mask position with numpy : 0.0004153251647949219 nb_pixel_total : 20016 time to create 1 rle with old method : 0.027683496475219727 length of segment : 212 time for calcul the mask position with numpy : 0.0009982585906982422 nb_pixel_total : 60318 time to create 1 rle with old method : 0.07525086402893066 length of segment : 236 time for calcul the mask position with numpy : 0.0002269744873046875 nb_pixel_total : 5971 time to create 1 rle with old method : 0.007675647735595703 length of segment : 143 time for calcul the mask position with numpy : 0.005307435989379883 nb_pixel_total : 287371 time to create 1 rle with new method : 0.04260063171386719 length of segment : 860 time for calcul the mask position with numpy : 0.0009224414825439453 nb_pixel_total : 51616 time to create 1 rle with old method : 0.07014083862304688 length of segment : 245 time for calcul the mask position with numpy : 0.00044798851013183594 nb_pixel_total : 9835 time to create 1 rle with old method : 0.01949763298034668 length of segment : 182 time for calcul the mask position with numpy : 0.0004935264587402344 nb_pixel_total : 25218 time to create 1 rle with old method : 0.03426790237426758 length of segment : 169 time for calcul the mask position with numpy : 0.0010712146759033203 nb_pixel_total : 40190 time to create 1 rle with old method : 0.0523076057434082 length of segment : 395 time for calcul the mask position with numpy : 0.0020859241485595703 nb_pixel_total : 72089 time to create 1 rle with old method : 0.11696505546569824 length of segment : 399 time for calcul the mask position with numpy : 0.00023484230041503906 nb_pixel_total : 4236 time to create 1 rle with old method : 0.0061800479888916016 length of segment : 174 time for calcul the mask position with numpy : 0.006137371063232422 nb_pixel_total : 289316 time to create 1 rle with new method : 0.0185549259185791 length of segment : 856 time for calcul the mask position with numpy : 0.002841949462890625 nb_pixel_total : 144756 time to create 1 rle with old method : 0.2154829502105713 length of segment : 420 time for calcul the mask position with numpy : 0.4546077251434326 nb_pixel_total : 2807414 time to create 1 rle with new method : 1.577763319015503 length of segment : 2956 time for calcul the mask position with numpy : 0.002123594284057617 nb_pixel_total : 60705 time to create 1 rle with old method : 0.06791424751281738 length of segment : 729 time for calcul the mask position with numpy : 0.0002655982971191406 nb_pixel_total : 2968 time to create 1 rle with old method : 0.003860950469970703 length of segment : 111 time for calcul the mask position with numpy : 0.0024476051330566406 nb_pixel_total : 125509 time to create 1 rle with old method : 0.14056110382080078 length of segment : 498 time for calcul the mask position with numpy : 0.0002722740173339844 nb_pixel_total : 4545 time to create 1 rle with old method : 0.00681757926940918 length of segment : 87 time for calcul the mask position with numpy : 0.018808364868164062 nb_pixel_total : 817938 time to create 1 rle with new method : 0.060105085372924805 length of segment : 1471 time for calcul the mask position with numpy : 0.024016141891479492 nb_pixel_total : 1284708 time to create 1 rle with new method : 0.22676992416381836 length of segment : 2052 time for calcul the mask position with numpy : 0.0021088123321533203 nb_pixel_total : 90691 time to create 1 rle with old method : 0.10564661026000977 length of segment : 374 time for calcul the mask position with numpy : 0.0013973712921142578 nb_pixel_total : 103948 time to create 1 rle with old method : 0.12521100044250488 length of segment : 451 time for calcul the mask position with numpy : 0.0021276473999023438 nb_pixel_total : 55813 time to create 1 rle with old method : 0.06454062461853027 length of segment : 436 time for calcul the mask position with numpy : 0.022006750106811523 nb_pixel_total : 800048 time to create 1 rle with new method : 0.22648954391479492 length of segment : 1506 time for calcul the mask position with numpy : 0.035286664962768555 nb_pixel_total : 1049948 time to create 1 rle with new method : 1.759627342224121 length of segment : 2273 time for calcul the mask position with numpy : 0.0021207332611083984 nb_pixel_total : 75961 time to create 1 rle with old method : 0.10974431037902832 length of segment : 330 time for calcul the mask position with numpy : 0.00026035308837890625 nb_pixel_total : 6333 time to create 1 rle with old method : 0.007560014724731445 length of segment : 118 time for calcul the mask position with numpy : 0.002246379852294922 nb_pixel_total : 103609 time to create 1 rle with old method : 0.11924958229064941 length of segment : 385 time for calcul the mask position with numpy : 0.0015225410461425781 nb_pixel_total : 53328 time to create 1 rle with old method : 0.07847833633422852 length of segment : 186 time for calcul the mask position with numpy : 0.0035600662231445312 nb_pixel_total : 185669 time to create 1 rle with new method : 0.008339166641235352 length of segment : 433 time for calcul the mask position with numpy : 0.0023674964904785156 nb_pixel_total : 168015 time to create 1 rle with new method : 0.005337953567504883 length of segment : 396 time for calcul the mask position with numpy : 0.0005431175231933594 nb_pixel_total : 18888 time to create 1 rle with old method : 0.025801420211791992 length of segment : 173 time for calcul the mask position with numpy : 0.005010843276977539 nb_pixel_total : 212104 time to create 1 rle with new method : 0.014604330062866211 length of segment : 913 time for calcul the mask position with numpy : 0.005545377731323242 nb_pixel_total : 249973 time to create 1 rle with new method : 0.013650655746459961 length of segment : 1006 time for calcul the mask position with numpy : 0.0067174434661865234 nb_pixel_total : 363198 time to create 1 rle with new method : 0.03075861930847168 length of segment : 1327 time for calcul the mask position with numpy : 0.005100250244140625 nb_pixel_total : 198314 time to create 1 rle with new method : 0.007683515548706055 length of segment : 679 time for calcul the mask position with numpy : 0.0005402565002441406 nb_pixel_total : 11068 time to create 1 rle with old method : 0.013748884201049805 length of segment : 152 time for calcul the mask position with numpy : 0.0010323524475097656 nb_pixel_total : 34172 time to create 1 rle with old method : 0.04036307334899902 length of segment : 276 time for calcul the mask position with numpy : 0.0006678104400634766 nb_pixel_total : 18952 time to create 1 rle with old method : 0.02323174476623535 length of segment : 185 time for calcul the mask position with numpy : 0.0034852027893066406 nb_pixel_total : 60926 time to create 1 rle with old method : 0.07198333740234375 length of segment : 556 time for calcul the mask position with numpy : 0.000782012939453125 nb_pixel_total : 27101 time to create 1 rle with old method : 0.03418445587158203 length of segment : 146 time for calcul the mask position with numpy : 0.0008327960968017578 nb_pixel_total : 22191 time to create 1 rle with old method : 0.028351545333862305 length of segment : 121 time for calcul the mask position with numpy : 0.0013954639434814453 nb_pixel_total : 42596 time to create 1 rle with old method : 0.05299234390258789 length of segment : 337 time for calcul the mask position with numpy : 0.02296757698059082 nb_pixel_total : 908887 time to create 1 rle with new method : 0.4409506320953369 length of segment : 1170 time for calcul the mask position with numpy : 0.0016705989837646484 nb_pixel_total : 53639 time to create 1 rle with old method : 0.06266379356384277 length of segment : 384 time for calcul the mask position with numpy : 0.0005166530609130859 nb_pixel_total : 19870 time to create 1 rle with old method : 0.025299072265625 length of segment : 195 time for calcul the mask position with numpy : 0.00012493133544921875 nb_pixel_total : 2017 time to create 1 rle with old method : 0.0024824142456054688 length of segment : 68 time for calcul the mask position with numpy : 0.00044417381286621094 nb_pixel_total : 8369 time to create 1 rle with old method : 0.010023355484008789 length of segment : 197 time for calcul the mask position with numpy : 0.017970561981201172 nb_pixel_total : 548439 time to create 1 rle with new method : 0.07180070877075195 length of segment : 943 time for calcul the mask position with numpy : 0.0039594173431396484 nb_pixel_total : 167094 time to create 1 rle with new method : 0.006309986114501953 length of segment : 404 time for calcul the mask position with numpy : 0.0067691802978515625 nb_pixel_total : 295126 time to create 1 rle with new method : 0.011358261108398438 length of segment : 507 time for calcul the mask position with numpy : 0.22299623489379883 nb_pixel_total : 1358378 time to create 1 rle with new method : 0.4760563373565674 length of segment : 2341 time for calcul the mask position with numpy : 0.005274534225463867 nb_pixel_total : 279715 time to create 1 rle with new method : 0.010439157485961914 length of segment : 509 time for calcul the mask position with numpy : 0.003401041030883789 nb_pixel_total : 158087 time to create 1 rle with new method : 0.007977008819580078 length of segment : 650 time for calcul the mask position with numpy : 0.0001251697540283203 nb_pixel_total : 1445 time to create 1 rle with old method : 0.0028493404388427734 length of segment : 56 time for calcul the mask position with numpy : 0.0049097537994384766 nb_pixel_total : 204496 time to create 1 rle with new method : 0.010387897491455078 length of segment : 675 time for calcul the mask position with numpy : 0.0010833740234375 nb_pixel_total : 35944 time to create 1 rle with old method : 0.042636871337890625 length of segment : 258 time for calcul the mask position with numpy : 0.9355707168579102 nb_pixel_total : 4206702 time to create 1 rle with new method : 0.8240926265716553 length of segment : 3960 time for calcul the mask position with numpy : 0.0002396106719970703 nb_pixel_total : 7300 time to create 1 rle with old method : 0.00895547866821289 length of segment : 153 time for calcul the mask position with numpy : 0.00011610984802246094 nb_pixel_total : 5595 time to create 1 rle with old method : 0.006751537322998047 length of segment : 86 time for calcul the mask position with numpy : 0.00024080276489257812 nb_pixel_total : 11361 time to create 1 rle with old method : 0.014012575149536133 length of segment : 217 time for calcul the mask position with numpy : 0.002543926239013672 nb_pixel_total : 209797 time to create 1 rle with new method : 0.006386280059814453 length of segment : 474 time for calcul the mask position with numpy : 0.0014567375183105469 nb_pixel_total : 114469 time to create 1 rle with old method : 0.1276230812072754 length of segment : 473 time for calcul the mask position with numpy : 0.004784107208251953 nb_pixel_total : 205963 time to create 1 rle with new method : 0.008174419403076172 length of segment : 644 time for calcul the mask position with numpy : 0.008621931076049805 nb_pixel_total : 324968 time to create 1 rle with new method : 0.014437675476074219 length of segment : 924 time for calcul the mask position with numpy : 0.04782748222351074 nb_pixel_total : 2265471 time to create 1 rle with new method : 0.2448287010192871 length of segment : 2166 time for calcul the mask position with numpy : 0.00019502639770507812 nb_pixel_total : 1701 time to create 1 rle with old method : 0.002269744873046875 length of segment : 85 time for calcul the mask position with numpy : 0.0009222030639648438 nb_pixel_total : 34964 time to create 1 rle with old method : 0.042691946029663086 length of segment : 592 time for calcul the mask position with numpy : 0.002069711685180664 nb_pixel_total : 57136 time to create 1 rle with old method : 0.07282471656799316 length of segment : 605 time for calcul the mask position with numpy : 0.001901865005493164 nb_pixel_total : 110624 time to create 1 rle with old method : 0.12806153297424316 length of segment : 454 time for calcul the mask position with numpy : 0.0008392333984375 nb_pixel_total : 47802 time to create 1 rle with old method : 0.0598905086517334 length of segment : 319 time for calcul the mask position with numpy : 0.006606340408325195 nb_pixel_total : 428448 time to create 1 rle with new method : 0.0159759521484375 length of segment : 1635 time for calcul the mask position with numpy : 0.0016598701477050781 nb_pixel_total : 124529 time to create 1 rle with old method : 0.1469271183013916 length of segment : 481 time for calcul the mask position with numpy : 0.0018146038055419922 nb_pixel_total : 119735 time to create 1 rle with old method : 0.14377498626708984 length of segment : 608 time for calcul the mask position with numpy : 0.003881216049194336 nb_pixel_total : 220201 time to create 1 rle with new method : 0.009796142578125 length of segment : 416 time for calcul the mask position with numpy : 0.003195524215698242 nb_pixel_total : 183808 time to create 1 rle with new method : 0.006551265716552734 length of segment : 449 time for calcul the mask position with numpy : 0.0005056858062744141 nb_pixel_total : 5629 time to create 1 rle with old method : 0.00702357292175293 length of segment : 117 time for calcul the mask position with numpy : 0.0007219314575195312 nb_pixel_total : 11757 time to create 1 rle with old method : 0.013925790786743164 length of segment : 213 time for calcul the mask position with numpy : 0.0006577968597412109 nb_pixel_total : 5742 time to create 1 rle with old method : 0.007241010665893555 length of segment : 174 time for calcul the mask position with numpy : 0.029984474182128906 nb_pixel_total : 871310 time to create 1 rle with new method : 0.08805394172668457 length of segment : 1157 time for calcul the mask position with numpy : 0.0016961097717285156 nb_pixel_total : 42207 time to create 1 rle with old method : 0.050786733627319336 length of segment : 333 time for calcul the mask position with numpy : 0.03422856330871582 nb_pixel_total : 1174486 time to create 1 rle with new method : 0.16107845306396484 length of segment : 1996 time for calcul the mask position with numpy : 0.0004162788391113281 nb_pixel_total : 9515 time to create 1 rle with old method : 0.011661767959594727 length of segment : 177 time for calcul the mask position with numpy : 0.00019931793212890625 nb_pixel_total : 2085 time to create 1 rle with old method : 0.0026159286499023438 length of segment : 98 time for calcul the mask position with numpy : 0.0004127025604248047 nb_pixel_total : 15072 time to create 1 rle with old method : 0.018405437469482422 length of segment : 532 time for calcul the mask position with numpy : 0.000164031982421875 nb_pixel_total : 5072 time to create 1 rle with old method : 0.006336212158203125 length of segment : 109 time for calcul the mask position with numpy : 0.0003643035888671875 nb_pixel_total : 2599 time to create 1 rle with old method : 0.0033261775970458984 length of segment : 68 time for calcul the mask position with numpy : 7.581710815429688e-05 nb_pixel_total : 1224 time to create 1 rle with old method : 0.0016446113586425781 length of segment : 34 time for calcul the mask position with numpy : 0.001964569091796875 nb_pixel_total : 75092 time to create 1 rle with old method : 0.09070730209350586 length of segment : 359 time for calcul the mask position with numpy : 0.0024619102478027344 nb_pixel_total : 115720 time to create 1 rle with old method : 0.13796329498291016 length of segment : 958 time for calcul the mask position with numpy : 0.0065267086029052734 nb_pixel_total : 427043 time to create 1 rle with new method : 0.015233278274536133 length of segment : 701 time for calcul the mask position with numpy : 0.0025005340576171875 nb_pixel_total : 75447 time to create 1 rle with old method : 0.09230780601501465 length of segment : 440 time for calcul the mask position with numpy : 0.0007212162017822266 nb_pixel_total : 16310 time to create 1 rle with old method : 0.025238990783691406 length of segment : 126 time for calcul the mask position with numpy : 0.0005834102630615234 nb_pixel_total : 4169 time to create 1 rle with old method : 0.0066225528717041016 length of segment : 174 time for calcul the mask position with numpy : 0.0062482357025146484 nb_pixel_total : 162855 time to create 1 rle with new method : 0.011359691619873047 length of segment : 393 time for calcul the mask position with numpy : 0.0032553672790527344 nb_pixel_total : 112732 time to create 1 rle with old method : 0.12927818298339844 length of segment : 431 time for calcul the mask position with numpy : 0.010843753814697266 nb_pixel_total : 411179 time to create 1 rle with new method : 0.01861119270324707 length of segment : 1115 time for calcul the mask position with numpy : 0.003888845443725586 nb_pixel_total : 100112 time to create 1 rle with old method : 0.12177896499633789 length of segment : 429 time for calcul the mask position with numpy : 0.001241445541381836 nb_pixel_total : 32548 time to create 1 rle with old method : 0.03978228569030762 length of segment : 350 time for calcul the mask position with numpy : 0.007664680480957031 nb_pixel_total : 303500 time to create 1 rle with new method : 0.014521360397338867 length of segment : 629 time for calcul the mask position with numpy : 0.0015647411346435547 nb_pixel_total : 65288 time to create 1 rle with old method : 0.07787394523620605 length of segment : 295 time for calcul the mask position with numpy : 0.0012660026550292969 nb_pixel_total : 35643 time to create 1 rle with old method : 0.04336190223693848 length of segment : 317 time for calcul the mask position with numpy : 0.0010318756103515625 nb_pixel_total : 30665 time to create 1 rle with old method : 0.03638124465942383 length of segment : 256 time for calcul the mask position with numpy : 0.0006129741668701172 nb_pixel_total : 28280 time to create 1 rle with old method : 0.03480029106140137 length of segment : 250 time for calcul the mask position with numpy : 0.011129379272460938 nb_pixel_total : 440632 time to create 1 rle with new method : 0.018915414810180664 length of segment : 788 time for calcul the mask position with numpy : 0.0031175613403320312 nb_pixel_total : 82244 time to create 1 rle with old method : 0.10034489631652832 length of segment : 482 time for calcul the mask position with numpy : 0.0016171932220458984 nb_pixel_total : 93083 time to create 1 rle with old method : 0.11087965965270996 length of segment : 425 time for calcul the mask position with numpy : 0.004344940185546875 nb_pixel_total : 169660 time to create 1 rle with new method : 0.008141517639160156 length of segment : 755 time for calcul the mask position with numpy : 0.0017087459564208984 nb_pixel_total : 63661 time to create 1 rle with old method : 0.07312774658203125 length of segment : 329 time for calcul the mask position with numpy : 0.0008664131164550781 nb_pixel_total : 35582 time to create 1 rle with old method : 0.04323077201843262 length of segment : 230 time for calcul the mask position with numpy : 0.14434266090393066 nb_pixel_total : 2505717 time to create 1 rle with new method : 0.1561892032623291 length of segment : 1765 time for calcul the mask position with numpy : 0.00064849853515625 nb_pixel_total : 23063 time to create 1 rle with old method : 0.02762770652770996 length of segment : 129 time for calcul the mask position with numpy : 0.0004456043243408203 nb_pixel_total : 16459 time to create 1 rle with old method : 0.02006363868713379 length of segment : 233 time spent for convertir_results : 30.666470527648926 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 125 chid ids of type : 3663 Number RLEs to save : 73438 save missing photos in datou_result : time spend for datou_step_exec : 78.6128888130188 time spend to save output : 6.14979887008667 total time spend for step 1 : 84.76268768310547 step2:crop_condition Fri Feb 28 01:45:56 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 125 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 14 About to insert : list_path_to_insert length 14 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 30 About to insert : list_path_to_insert length 30 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1339998274, 1339998272, 1339998237, 1339959122, 1339959044, 1339958960, 1339958851, 1339958849, 1339958847, 1339958697] Looping around the photos to save general results len do output : 66 /-3689329782Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329796Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329798Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329795Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329820Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329829Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329832Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329850Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329857Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329791Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329787Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329801Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329805Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329815Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329821Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329838Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329844Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329854Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329864Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329870Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329900Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329894Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329906Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329785Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329827Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329849Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329855Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329869Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329883Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329889Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329895Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329891Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329788Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329793Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329786Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329790Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329794Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329806Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329799Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329816Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329822Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329823Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329846Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329833Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329847Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329843Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329856Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329862Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329863Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329867Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329866Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329882Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329885Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329874Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329872Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329892Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329896Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329888Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329893Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329905Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329901Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329903Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329830Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329902Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329802Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689329814Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998274', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998272', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998237', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959122', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959044', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958960', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958851', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958849', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958847', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958697', None, None, None, None, None, '2621501') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 208 time used for this insertion : 0.04205179214477539 save_final save missing photos in datou_result : time spend for datou_step_exec : 41.701820850372314 time spend to save output : 0.044187307357788086 total time spend for step 2 : 41.7460081577301 step3:thcl Fri Feb 28 01:46:38 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.007425546646118164 time to convert the images to numpy array : 0.11843347549438477 time to import caffe and check if the image exist : 0.01264643669128418 time to convert the images to numpy array : 0.4721083641052246 time to import caffe and check if the image exist : 0.0075376033782958984 time to convert the images to numpy array : 0.4898688793182373 time to import caffe and check if the image exist : 0.011017560958862305 time to convert the images to numpy array : 0.48952269554138184 time to import caffe and check if the image exist : 0.01609182357788086 time to convert the images to numpy array : 0.8790292739868164 time to import caffe and check if the image exist : 0.013876914978027344 time to convert the images to numpy array : 0.8882343769073486 time to import caffe and check if the image exist : 0.008905172348022461 time to convert the images to numpy array : 0.8938078880310059 time to import caffe and check if the image exist : 0.015787839889526367 time to convert the images to numpy array : 0.8992972373962402 time to import caffe and check if the image exist : 0.014198064804077148 time to convert the images to numpy array : 0.9261744022369385 time to import caffe and check if the image exist : 0.010393142700195312 time to convert the images to numpy array : 1.629251480102539 total time to convert the images to numpy array : 1.9025461673736572 list photo_ids error: [] list photo_ids correct : [-3689329902, -3689329802, -3689329814, -3689329821, -3689329838, -3689329844, -3689329854, -3689329864, -3689329870, -3689329900, -3689329782, -3689329796, -3689329798, -3689329795, -3689329820, -3689329829, -3689329832, -3689329867, -3689329866, -3689329882, -3689329885, -3689329874, -3689329872, -3689329892, -3689329896, -3689329888, -3689329893, -3689329905, -3689329901, -3689329903, -3689329830, -3689329846, -3689329833, -3689329847, -3689329843, -3689329856, -3689329862, -3689329863, -3689329850, -3689329857, -3689329791, -3689329787, -3689329801, -3689329805, -3689329815, -3689329894, -3689329906, -3689329785, -3689329827, -3689329849, -3689329855, -3689329869, -3689329790, -3689329794, -3689329806, -3689329799, -3689329816, -3689329822, -3689329823, -3689329883, -3689329889, -3689329895, -3689329891, -3689329788, -3689329793, -3689329786] number of photos to traite : 66 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5304 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5085 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.9881534576416016 time used to do the prediction : 0.18974542617797852 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.307065725326538 time used to do the prediction : 0.14357829093933105 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.2874603271484375e-05 save missing photos in datou_result : time spend for datou_step_exec : 11.386680841445923 time spend to save output : 0.0005095005035400391 total time spend for step 3 : 11.387190341949463 step4:merge_mask_thcl_custom Fri Feb 28 01:46:49 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 125 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1339998274, 1339998272, 1339998237, 1339959122, 1339959044, 1339958960, 1339958851, 1339958849, 1339958847, 1339958697] Looping around the photos to save general results len do output : 10 /1339998274Didn't retrieve data .Didn't retrieve data . /1339998272Didn't retrieve data .Didn't retrieve data . /1339998237Didn't retrieve data .Didn't retrieve data . /1339959122Didn't retrieve data .Didn't retrieve data . /1339959044Didn't retrieve data .Didn't retrieve data . /1339958960Didn't retrieve data .Didn't retrieve data . /1339958851Didn't retrieve data .Didn't retrieve data . /1339958849Didn't retrieve data .Didn't retrieve data . /1339958847Didn't retrieve data .Didn't retrieve data . /1339958697Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998274', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998272', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998237', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959122', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959044', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958960', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958851', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958849', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958847', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958697', None, None, None, None, None, '2621501') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.2211928367614746 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.3086249828338623 time spend to save output : 0.2216358184814453 total time spend for step 4 : 0.5302608013153076 step5:rle_unique_nms_with_priority Fri Feb 28 01:46:50 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 12 nb_hashtags : 5 time to prepare the origin masks : 15.740030288696289 time for calcul the mask position with numpy : 0.18049931526184082 nb_pixel_total : 4617577 time to create 1 rle with new method : 0.713092565536499 time for calcul the mask position with numpy : 0.03411149978637695 nb_pixel_total : 6155 time to create 1 rle with old method : 0.007066249847412109 time for calcul the mask position with numpy : 0.034241437911987305 nb_pixel_total : 11136 time to create 1 rle with old method : 0.012681961059570312 time for calcul the mask position with numpy : 0.030962705612182617 nb_pixel_total : 57489 time to create 1 rle with old method : 0.06777834892272949 time for calcul the mask position with numpy : 0.022174596786499023 nb_pixel_total : 2314 time to create 1 rle with old method : 0.0026526451110839844 time for calcul the mask position with numpy : 0.02184462547302246 nb_pixel_total : 3179 time to create 1 rle with old method : 0.0037746429443359375 time for calcul the mask position with numpy : 0.021962404251098633 nb_pixel_total : 27488 time to create 1 rle with old method : 0.030797481536865234 time for calcul the mask position with numpy : 0.021931886672973633 nb_pixel_total : 27395 time to create 1 rle with old method : 0.030939340591430664 time for calcul the mask position with numpy : 0.02074432373046875 nb_pixel_total : 4840 time to create 1 rle with old method : 0.005937099456787109 time for calcul the mask position with numpy : 0.027335643768310547 nb_pixel_total : 668252 time to create 1 rle with new method : 0.8568484783172607 time for calcul the mask position with numpy : 0.03779292106628418 nb_pixel_total : 1490298 time to create 1 rle with new method : 0.7946772575378418 time for calcul the mask position with numpy : 0.021785259246826172 nb_pixel_total : 56917 time to create 1 rle with old method : 0.06705665588378906 time for calcul the mask position with numpy : 0.022693872451782227 nb_pixel_total : 77200 time to create 1 rle with old method : 0.09435105323791504 create new chi : 3.26359224319458 time to delete rle : 0.019751787185668945 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 13120 TO DO : save crop sub photo not yet done ! save time : 1.35221266746521 nb_obj : 12 nb_hashtags : 6 time to prepare the origin masks : 14.291097164154053 time for calcul the mask position with numpy : 0.7756130695343018 nb_pixel_total : 6327940 time to create 1 rle with new method : 0.5286533832550049 time for calcul the mask position with numpy : 0.03506755828857422 nb_pixel_total : 12698 time to create 1 rle with old method : 0.014536619186401367 time for calcul the mask position with numpy : 0.034609317779541016 nb_pixel_total : 59418 time to create 1 rle with old method : 0.06682038307189941 time for calcul the mask position with numpy : 0.037908315658569336 nb_pixel_total : 288861 time to create 1 rle with new method : 0.5883538722991943 time for calcul the mask position with numpy : 0.03449726104736328 nb_pixel_total : 51484 time to create 1 rle with old method : 0.05723881721496582 time for calcul the mask position with numpy : 0.028050661087036133 nb_pixel_total : 4220 time to create 1 rle with old method : 0.00480198860168457 time for calcul the mask position with numpy : 0.02632617950439453 nb_pixel_total : 40080 time to create 1 rle with old method : 0.0634005069732666 time for calcul the mask position with numpy : 0.023036956787109375 nb_pixel_total : 19904 time to create 1 rle with old method : 0.021840810775756836 time for calcul the mask position with numpy : 0.021527767181396484 nb_pixel_total : 9769 time to create 1 rle with old method : 0.011148929595947266 time for calcul the mask position with numpy : 0.021215438842773438 nb_pixel_total : 60123 time to create 1 rle with old method : 0.06490135192871094 time for calcul the mask position with numpy : 0.020274639129638672 nb_pixel_total : 5957 time to create 1 rle with old method : 0.00654911994934082 time for calcul the mask position with numpy : 0.022123098373413086 nb_pixel_total : 25174 time to create 1 rle with old method : 0.028063535690307617 time for calcul the mask position with numpy : 0.033113718032836914 nb_pixel_total : 144612 time to create 1 rle with old method : 0.1674199104309082 create new chi : 2.798029661178589 time to delete rle : 0.0010192394256591797 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 10048 TO DO : save crop sub photo not yet done ! save time : 0.6465981006622314 nb_obj : 10 nb_hashtags : 3 time to prepare the origin masks : 7.516263008117676 time for calcul the mask position with numpy : 0.3088715076446533 nb_pixel_total : 3476817 time to create 1 rle with new method : 0.691272497177124 time for calcul the mask position with numpy : 0.0374908447265625 nb_pixel_total : 100 time to create 1 rle with old method : 0.00021910667419433594 time for calcul the mask position with numpy : 0.04108119010925293 nb_pixel_total : 90559 time to create 1 rle with old method : 0.09889626502990723 time for calcul the mask position with numpy : 0.03443551063537598 nb_pixel_total : 55724 time to create 1 rle with old method : 0.059883832931518555 time for calcul the mask position with numpy : 0.03772616386413574 nb_pixel_total : 404462 time to create 1 rle with new method : 0.4729294776916504 time for calcul the mask position with numpy : 0.03160595893859863 nb_pixel_total : 77069 time to create 1 rle with old method : 0.0963442325592041 time for calcul the mask position with numpy : 0.03380703926086426 nb_pixel_total : 4504 time to create 1 rle with old method : 0.007031679153442383 time for calcul the mask position with numpy : 0.029730558395385742 nb_pixel_total : 90185 time to create 1 rle with old method : 0.09923911094665527 time for calcul the mask position with numpy : 0.021647930145263672 nb_pixel_total : 2958 time to create 1 rle with old method : 0.0033423900604248047 time for calcul the mask position with numpy : 0.020943403244018555 nb_pixel_total : 59538 time to create 1 rle with old method : 0.06683492660522461 time for calcul the mask position with numpy : 0.25444960594177246 nb_pixel_total : 2788324 time to create 1 rle with new method : 0.8288116455078125 create new chi : 3.3620998859405518 time to delete rle : 0.0014107227325439453 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 15227 TO DO : save crop sub photo not yet done ! save time : 1.2567951679229736 nb_obj : 10 nb_hashtags : 6 time to prepare the origin masks : 14.282503843307495 time for calcul the mask position with numpy : 0.5352175235748291 nb_pixel_total : 4403184 time to create 1 rle with new method : 1.215489149093628 time for calcul the mask position with numpy : 0.035019636154174805 nb_pixel_total : 137669 time to create 1 rle with old method : 0.17367291450500488 time for calcul the mask position with numpy : 0.03396248817443848 nb_pixel_total : 103481 time to create 1 rle with old method : 0.11449527740478516 time for calcul the mask position with numpy : 0.03469204902648926 nb_pixel_total : 3560 time to create 1 rle with old method : 0.004735708236694336 time for calcul the mask position with numpy : 0.038890838623046875 nb_pixel_total : 184876 time to create 1 rle with new method : 1.9363360404968262 time for calcul the mask position with numpy : 0.020806312561035156 nb_pixel_total : 6311 time to create 1 rle with old method : 0.0070590972900390625 time for calcul the mask position with numpy : 0.022213220596313477 nb_pixel_total : 53242 time to create 1 rle with old method : 0.06542110443115234 time for calcul the mask position with numpy : 0.02204442024230957 nb_pixel_total : 75764 time to create 1 rle with old method : 0.08163714408874512 time for calcul the mask position with numpy : 0.03292536735534668 nb_pixel_total : 1012124 time to create 1 rle with new method : 1.6100835800170898 time for calcul the mask position with numpy : 0.022570371627807617 nb_pixel_total : 272649 time to create 1 rle with new method : 1.491562843322754 time for calcul the mask position with numpy : 0.042404890060424805 nb_pixel_total : 797380 time to create 1 rle with new method : 0.9772098064422607 create new chi : 8.647009134292603 time to delete rle : 0.0021982192993164062 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 15513 TO DO : save crop sub photo not yet done ! save time : 1.329031229019165 nb_obj : 21 nb_hashtags : 8 time to prepare the origin masks : 21.97620940208435 time for calcul the mask position with numpy : 0.7872781753540039 nb_pixel_total : 4292461 time to create 1 rle with new method : 0.9714508056640625 time for calcul the mask position with numpy : 0.038060665130615234 nb_pixel_total : 18920 time to create 1 rle with old method : 0.020920515060424805 time for calcul the mask position with numpy : 0.036988019943237305 nb_pixel_total : 1431 time to create 1 rle with old method : 0.002120494842529297 time for calcul the mask position with numpy : 0.040001630783081055 nb_pixel_total : 174513 time to create 1 rle with new method : 0.6309211254119873 time for calcul the mask position with numpy : 0.034796953201293945 nb_pixel_total : 60471 time to create 1 rle with old method : 0.0694279670715332 time for calcul the mask position with numpy : 0.03484654426574707 nb_pixel_total : 7107 time to create 1 rle with old method : 0.008752107620239258 time for calcul the mask position with numpy : 0.03384256362915039 nb_pixel_total : 11029 time to create 1 rle with old method : 0.012885093688964844 time for calcul the mask position with numpy : 0.03533291816711426 nb_pixel_total : 166694 time to create 1 rle with new method : 0.6432147026062012 time for calcul the mask position with numpy : 0.0430140495300293 nb_pixel_total : 547832 time to create 1 rle with new method : 0.7620120048522949 time for calcul the mask position with numpy : 0.03769493103027344 nb_pixel_total : 2054 time to create 1 rle with old method : 0.0031616687774658203 time for calcul the mask position with numpy : 0.03757667541503906 nb_pixel_total : 8369 time to create 1 rle with old method : 0.011791229248046875 time for calcul the mask position with numpy : 0.04202628135681152 nb_pixel_total : 197400 time to create 1 rle with new method : 0.9441318511962891 time for calcul the mask position with numpy : 0.037302494049072266 nb_pixel_total : 294186 time to create 1 rle with new method : 0.6156647205352783 time for calcul the mask position with numpy : 0.033310651779174805 nb_pixel_total : 1999 time to create 1 rle with old method : 0.0023953914642333984 time for calcul the mask position with numpy : 0.03363323211669922 nb_pixel_total : 19750 time to create 1 rle with old method : 0.022923707962036133 time for calcul the mask position with numpy : 0.0386350154876709 nb_pixel_total : 157839 time to create 1 rle with new method : 0.7781007289886475 time for calcul the mask position with numpy : 0.045836687088012695 nb_pixel_total : 908409 time to create 1 rle with new method : 0.43018674850463867 time for calcul the mask position with numpy : 0.03747296333312988 nb_pixel_total : 42555 time to create 1 rle with old method : 0.05241250991821289 time for calcul the mask position with numpy : 0.042154550552368164 nb_pixel_total : 22071 time to create 1 rle with old method : 0.03318166732788086 time for calcul the mask position with numpy : 0.04089641571044922 nb_pixel_total : 27201 time to create 1 rle with old method : 0.030719757080078125 time for calcul the mask position with numpy : 0.037657976150512695 nb_pixel_total : 34046 time to create 1 rle with old method : 0.03756213188171387 time for calcul the mask position with numpy : 0.03400421142578125 nb_pixel_total : 53903 time to create 1 rle with old method : 0.059525251388549805 create new chi : 7.934219837188721 time to delete rle : 0.002767324447631836 batch 1 Loaded 22 chid ids of type : 3726 Number RLEs to save : 18581 TO DO : save crop sub photo not yet done ! save time : 1.6442487239837646 nb_obj : 6 nb_hashtags : 4 time to prepare the origin masks : 6.587132930755615 time for calcul the mask position with numpy : 0.07404208183288574 nb_pixel_total : 2421934 time to create 1 rle with new method : 1.260087490081787 time for calcul the mask position with numpy : 0.038421630859375 nb_pixel_total : 5552 time to create 1 rle with old method : 0.0058422088623046875 time for calcul the mask position with numpy : 0.0399322509765625 nb_pixel_total : 113590 time to create 1 rle with old method : 0.12103271484375 time for calcul the mask position with numpy : 0.03437399864196777 nb_pixel_total : 35888 time to create 1 rle with old method : 0.041229963302612305 time for calcul the mask position with numpy : 0.035257816314697266 nb_pixel_total : 7265 time to create 1 rle with old method : 0.007978677749633789 time for calcul the mask position with numpy : 0.019634485244750977 nb_pixel_total : 11310 time to create 1 rle with old method : 0.012229204177856445 time for calcul the mask position with numpy : 0.47966909408569336 nb_pixel_total : 4454701 time to create 1 rle with new method : 0.613785982131958 create new chi : 2.847550392150879 time to delete rle : 0.0014774799346923828 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 10770 TO DO : save crop sub photo not yet done ! save time : 0.6775248050689697 nb_obj : 14 nb_hashtags : 5 time to prepare the origin masks : 13.56407117843628 time for calcul the mask position with numpy : 1.8418340682983398 nb_pixel_total : 3473827 time to create 1 rle with new method : 1.4750938415527344 time for calcul the mask position with numpy : 0.034850358963012695 nb_pixel_total : 47665 time to create 1 rle with old method : 0.05206155776977539 time for calcul the mask position with numpy : 0.035872697830200195 nb_pixel_total : 205798 time to create 1 rle with new method : 1.6653578281402588 time for calcul the mask position with numpy : 0.038847923278808594 nb_pixel_total : 56920 time to create 1 rle with old method : 0.06292319297790527 time for calcul the mask position with numpy : 0.0341641902923584 nb_pixel_total : 60001 time to create 1 rle with old method : 0.07133007049560547 time for calcul the mask position with numpy : 0.03427481651306152 nb_pixel_total : 68220 time to create 1 rle with old method : 0.07887029647827148 time for calcul the mask position with numpy : 0.03499627113342285 nb_pixel_total : 91901 time to create 1 rle with old method : 0.11003541946411133 time for calcul the mask position with numpy : 0.03476262092590332 nb_pixel_total : 110373 time to create 1 rle with old method : 0.12004256248474121 time for calcul the mask position with numpy : 0.0339663028717041 nb_pixel_total : 5390 time to create 1 rle with old method : 0.00646209716796875 time for calcul the mask position with numpy : 0.03998374938964844 nb_pixel_total : 34928 time to create 1 rle with old method : 0.04101824760437012 time for calcul the mask position with numpy : 0.034558773040771484 nb_pixel_total : 1688 time to create 1 rle with old method : 0.0019235610961914062 time for calcul the mask position with numpy : 0.1581711769104004 nb_pixel_total : 2262556 time to create 1 rle with new method : 1.7856764793395996 time for calcul the mask position with numpy : 0.03786349296569824 nb_pixel_total : 323768 time to create 1 rle with new method : 1.4270801544189453 time for calcul the mask position with numpy : 0.03447365760803223 nb_pixel_total : 123860 time to create 1 rle with old method : 0.13270878791809082 time for calcul the mask position with numpy : 0.035241127014160156 nb_pixel_total : 183345 time to create 1 rle with new method : 1.649946928024292 create new chi : 11.27022099494934 time to delete rle : 0.0025017261505126953 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 17813 TO DO : save crop sub photo not yet done ! save time : 2.3751254081726074 nb_obj : 16 nb_hashtags : 5 time to prepare the origin masks : 15.81326675415039 time for calcul the mask position with numpy : 0.8346867561340332 nb_pixel_total : 4768654 time to create 1 rle with new method : 1.0249791145324707 time for calcul the mask position with numpy : 0.03366231918334961 nb_pixel_total : 42110 time to create 1 rle with old method : 0.0464932918548584 time for calcul the mask position with numpy : 0.0209352970123291 nb_pixel_total : 75084 time to create 1 rle with old method : 0.0826575756072998 time for calcul the mask position with numpy : 0.02249765396118164 nb_pixel_total : 4135 time to create 1 rle with old method : 0.004792213439941406 time for calcul the mask position with numpy : 0.022243261337280273 nb_pixel_total : 5741 time to create 1 rle with old method : 0.006597280502319336 time for calcul the mask position with numpy : 0.023947954177856445 nb_pixel_total : 79214 time to create 1 rle with old method : 0.08697009086608887 time for calcul the mask position with numpy : 0.02221059799194336 nb_pixel_total : 1215 time to create 1 rle with old method : 0.001397848129272461 time for calcul the mask position with numpy : 0.021730899810791016 nb_pixel_total : 2574 time to create 1 rle with old method : 0.0031270980834960938 time for calcul the mask position with numpy : 0.02153468132019043 nb_pixel_total : 5051 time to create 1 rle with old method : 0.0072269439697265625 time for calcul the mask position with numpy : 0.02335333824157715 nb_pixel_total : 15017 time to create 1 rle with old method : 0.02400994300842285 time for calcul the mask position with numpy : 0.023164749145507812 nb_pixel_total : 2075 time to create 1 rle with old method : 0.0034515857696533203 time for calcul the mask position with numpy : 0.0220181941986084 nb_pixel_total : 9478 time to create 1 rle with old method : 0.010209321975708008 time for calcul the mask position with numpy : 0.03354239463806152 nb_pixel_total : 958794 time to create 1 rle with new method : 1.1547632217407227 time for calcul the mask position with numpy : 0.026471853256225586 nb_pixel_total : 871074 time to create 1 rle with new method : 0.5092923641204834 time for calcul the mask position with numpy : 0.02262568473815918 nb_pixel_total : 8916 time to create 1 rle with old method : 0.009987831115722656 time for calcul the mask position with numpy : 0.024132966995239258 nb_pixel_total : 75044 time to create 1 rle with old method : 0.08801984786987305 time for calcul the mask position with numpy : 0.02614760398864746 nb_pixel_total : 126064 time to create 1 rle with old method : 0.1482391357421875 create new chi : 4.522462844848633 time to delete rle : 0.0014333724975585938 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 14842 TO DO : save crop sub photo not yet done ! save time : 1.349672794342041 nb_obj : 11 nb_hashtags : 6 time to prepare the origin masks : 11.183578968048096 time for calcul the mask position with numpy : 0.9036891460418701 nb_pixel_total : 5376814 time to create 1 rle with new method : 1.833611011505127 time for calcul the mask position with numpy : 0.03197956085205078 nb_pixel_total : 7350 time to create 1 rle with old method : 0.008014202117919922 time for calcul the mask position with numpy : 0.020139217376708984 nb_pixel_total : 8334 time to create 1 rle with old method : 0.009721994400024414 time for calcul the mask position with numpy : 0.02265787124633789 nb_pixel_total : 440888 time to create 1 rle with new method : 1.3541004657745361 time for calcul the mask position with numpy : 0.0193936824798584 nb_pixel_total : 32506 time to create 1 rle with old method : 0.034009695053100586 time for calcul the mask position with numpy : 0.019663095474243164 nb_pixel_total : 162373 time to create 1 rle with new method : 1.4263296127319336 time for calcul the mask position with numpy : 0.020493507385253906 nb_pixel_total : 30436 time to create 1 rle with old method : 0.0327000617980957 time for calcul the mask position with numpy : 0.020171403884887695 nb_pixel_total : 65219 time to create 1 rle with old method : 0.0720510482788086 time for calcul the mask position with numpy : 0.024220705032348633 nb_pixel_total : 417585 time to create 1 rle with new method : 0.9611556529998779 time for calcul the mask position with numpy : 0.020351886749267578 nb_pixel_total : 92940 time to create 1 rle with old method : 0.09818124771118164 time for calcul the mask position with numpy : 0.02115941047668457 nb_pixel_total : 303108 time to create 1 rle with new method : 0.49536967277526855 time for calcul the mask position with numpy : 0.021855831146240234 nb_pixel_total : 112687 time to create 1 rle with old method : 0.11884546279907227 create new chi : 7.709187269210815 time to delete rle : 0.0015721321105957031 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 12378 TO DO : save crop sub photo not yet done ! save time : 1.037914514541626 nb_obj : 6 nb_hashtags : 5 time to prepare the origin masks : 7.071728229522705 time for calcul the mask position with numpy : 0.19867300987243652 nb_pixel_total : 4237087 time to create 1 rle with new method : 1.1251904964447021 time for calcul the mask position with numpy : 0.02063441276550293 nb_pixel_total : 35505 time to create 1 rle with old method : 0.04072141647338867 time for calcul the mask position with numpy : 0.025633573532104492 nb_pixel_total : 169329 time to create 1 rle with new method : 0.8167293071746826 time for calcul the mask position with numpy : 0.020582199096679688 nb_pixel_total : 16426 time to create 1 rle with old method : 0.017936229705810547 time for calcul the mask position with numpy : 0.020865440368652344 nb_pixel_total : 22959 time to create 1 rle with old method : 0.025604963302612305 time for calcul the mask position with numpy : 0.06344985961914062 nb_pixel_total : 2505435 time to create 1 rle with new method : 0.923431396484375 time for calcul the mask position with numpy : 0.02135157585144043 nb_pixel_total : 63499 time to create 1 rle with old method : 0.0702064037322998 create new chi : 3.4701497554779053 time to delete rle : 0.0008192062377929688 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 9062 TO DO : save crop sub photo not yet done ! save time : 0.635749340057373 map_output_result : {1339998274: (0.0, 'Should be the crop_list due to order', 0), 1339998272: (0.0, 'Should be the crop_list due to order', 0), 1339998237: (0.0, 'Should be the crop_list due to order', 0), 1339959122: (0.0, 'Should be the crop_list due to order', 0), 1339959044: (0.0, 'Should be the crop_list due to order', 0), 1339958960: (0.0, 'Should be the crop_list due to order', 0), 1339958851: (0.0, 'Should be the crop_list due to order', 0), 1339958849: (0.0, 'Should be the crop_list due to order', 0), 1339958847: (0.0, 'Should be the crop_list due to order', 0), 1339958697: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1339998274, 1339998272, 1339998237, 1339959122, 1339959044, 1339958960, 1339958851, 1339958849, 1339958847, 1339958697] Looping around the photos to save general results len do output : 10 /1339998274.Didn't retrieve data . /1339998272.Didn't retrieve data . /1339998237.Didn't retrieve data . /1339959122.Didn't retrieve data . /1339959044.Didn't retrieve data . /1339958960.Didn't retrieve data . /1339958851.Didn't retrieve data . /1339958849.Didn't retrieve data . /1339958847.Didn't retrieve data . /1339958697.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998274', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998272', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998237', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959122', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959044', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958960', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958851', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958849', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958847', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958697', None, None, None, None, None, '2621501') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014505863189697266 save_final save missing photos in datou_result : time spend for datou_step_exec : 197.41759252548218 time spend to save output : 0.015188217163085938 total time spend for step 5 : 197.43278074264526 step6:crop_condition Fri Feb 28 01:50:07 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 128 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740703811_2350145 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.5286285877227783 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 4869462 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740703822_2350145 we have uploaded 9 photos in the portfolio 4869462 time of upload the photos Elapsed time : 3.0606253147125244 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740703829_2350145 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.5309226512908936 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740703834_2350145 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8936388492584229 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740703837_2350145 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.0414621829986572 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740703842_2350145 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.030791997909546 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1339998274, 1339998272, 1339998237, 1339959122, 1339959044, 1339958960, 1339958851, 1339958849, 1339958847, 1339958697] Looping around the photos to save general results len do output : 27 /1340029511Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029512Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029513Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029514Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029579Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029581Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029583Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029585Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029587Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029589Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029591Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029593Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029595Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029693Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029696Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029698Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029778Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029780Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029782Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029848Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029849Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029850Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029851Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029852Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029853Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029858Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340029859Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998274', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998272', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998237', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959122', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959044', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958960', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958851', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958849', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958847', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958697', None, None, None, None, None, '2621501') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 91 time used for this insertion : 0.15316390991210938 save_final save missing photos in datou_result : time spend for datou_step_exec : 36.26974058151245 time spend to save output : 0.15410280227661133 total time spend for step 6 : 36.42384338378906 step7:ventilate_hashtags_in_portfolio Fri Feb 28 01:50:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20901284 get user id for portfolio 20901284 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901284 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_brun','Teint_Dans_La_Masse','papier','plastique','cartonnette','kraft','metal','environnement','flou','autre_refus','Carton_gris','mal_croppe')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901284 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_brun','Teint_Dans_La_Masse','papier','plastique','cartonnette','kraft','metal','environnement','flou','autre_refus','Carton_gris','mal_croppe')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901284 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_brun','Teint_Dans_La_Masse','papier','plastique','cartonnette','kraft','metal','environnement','flou','autre_refus','Carton_gris','mal_croppe')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20902359,20902360,20902361,20902362,20902363,20902364,20902365,20902366,20902367,20902368,20902369,20902370?tags=Carton_brun,Teint_Dans_La_Masse,papier,plastique,cartonnette,kraft,metal,environnement,flou,autre_refus,Carton_gris,mal_croppe Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1339998274, 1339998272, 1339998237, 1339959122, 1339959044, 1339958960, 1339958851, 1339958849, 1339958847, 1339958697] Looping around the photos to save general results len do output : 1 /20901284. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998274', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998272', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998237', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959122', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959044', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958960', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958851', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958849', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958847', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958697', None, None, None, None, None, '2621501') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.017199993133544922 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.011512041091919 time spend to save output : 0.0179140567779541 total time spend for step 7 : 2.029426097869873 step8:final Fri Feb 28 01:50:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1339998274: ('0.1199733468175425',), 1339998272: ('0.1199733468175425',), 1339998237: ('0.1199733468175425',), 1339959122: ('0.1199733468175425',), 1339959044: ('0.1199733468175425',), 1339958960: ('0.1199733468175425',), 1339958851: ('0.1199733468175425',), 1339958849: ('0.1199733468175425',), 1339958847: ('0.1199733468175425',), 1339958697: ('0.1199733468175425',)} new output for save of step final : {1339998274: ('0.1199733468175425',), 1339998272: ('0.1199733468175425',), 1339998237: ('0.1199733468175425',), 1339959122: ('0.1199733468175425',), 1339959044: ('0.1199733468175425',), 1339958960: ('0.1199733468175425',), 1339958851: ('0.1199733468175425',), 1339958849: ('0.1199733468175425',), 1339958847: ('0.1199733468175425',), 1339958697: ('0.1199733468175425',)} [1339998274, 1339998272, 1339998237, 1339959122, 1339959044, 1339958960, 1339958851, 1339958849, 1339958847, 1339958697] Looping around the photos to save general results len do output : 10 /1339998274.Didn't retrieve data . /1339998272.Didn't retrieve data . /1339998237.Didn't retrieve data . /1339959122.Didn't retrieve data . /1339959044.Didn't retrieve data . /1339958960.Didn't retrieve data . /1339958851.Didn't retrieve data . /1339958849.Didn't retrieve data . /1339958847.Didn't retrieve data . /1339958697.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998274', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998272', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998237', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959122', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959044', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958960', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958851', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958849', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958847', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958697', None, None, None, None, None, '2621501') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.015177488327026367 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11357712745666504 time spend to save output : 0.015705585479736328 total time spend for step 8 : 0.12928271293640137 step9:velours_tree Fri Feb 28 01:50:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.028612375259399414 time spend to save output : 2.956390380859375e-05 total time spend for step 9 : 0.028641939163208008 step10:send_mail_cod Fri Feb 28 01:50:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20901284_28-02-2025_01_50_46.pdf 20902362 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209023621740703846 20902363 imagette209023631740703846 20902364 change filename to text .change filename to text .imagette209023641740703846 20902365 imagette209023651740703846 20902367 imagette209023671740703846 20902368 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209023681740703846 20902370 imagette209023701740703847 20902359 change filename to text .change filename to text .change filename to text .imagette209023591740703847 20902360 change filename to text .change filename to text .change filename to text .change filename to text .imagette209023601740703847 20902369 change filename to text .change filename to text .change filename to text .imagette209023691740703848 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20901284 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20902359,20902360,20902361,20902362,20902363,20902364,20902365,20902366,20902367,20902368,20902369,20902370?tags=Carton_brun,Teint_Dans_La_Masse,papier,plastique,cartonnette,kraft,metal,environnement,flou,autre_refus,Carton_gris,mal_croppe your option no_mail is active, we will not send the real mail to your client args[1339998274] : ('0.1199733468175425',) no score found for photo 1339998274 We are sending mail with results at report@fotonower.com args[1339998272] : ('0.1199733468175425',) no score found for photo 1339998272 We are sending mail with results at report@fotonower.com args[1339998237] : ('0.1199733468175425',) no score found for photo 1339998237 We are sending mail with results at report@fotonower.com args[1339959122] : ('0.1199733468175425',) no score found for photo 1339959122 We are sending mail with results at report@fotonower.com args[1339959044] : ('0.1199733468175425',) no score found for photo 1339959044 We are sending mail with results at report@fotonower.com args[1339958960] : ('0.1199733468175425',) no score found for photo 1339958960 We are sending mail with results at report@fotonower.com args[1339958851] : ('0.1199733468175425',) no score found for photo 1339958851 We are sending mail with results at report@fotonower.com args[1339958849] : ('0.1199733468175425',) no score found for photo 1339958849 We are sending mail with results at report@fotonower.com args[1339958847] : ('0.1199733468175425',) no score found for photo 1339958847 We are sending mail with results at report@fotonower.com args[1339958697] : ('0.1199733468175425',) no score found for photo 1339958697 We are sending mail with results at report@fotonower.com refus_total : 0.1199733468175425 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20901284 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1339958294,1339958296,1339958316,1339958363,1339958849,1339959044,1339998272,1339958527,1339958697,1339958847,1339958851,1339958960,1339959122,1339998237,1339998274) Found this number of photos: 15 begin to download photo : 1339958294 begin to download photo : 1339958849 begin to download photo : 1339958697 begin to download photo : 1339959122 download finish for photo 1339958849 begin to download photo : 1339959044 download finish for photo 1339958294 begin to download photo : 1339958296 download finish for photo 1339959122 begin to download photo : 1339998237 download finish for photo 1339958697 begin to download photo : 1339958847 download finish for photo 1339959044 begin to download photo : 1339998272 download finish for photo 1339958296 begin to download photo : 1339958316 download finish for photo 1339998237 begin to download photo : 1339998274 download finish for photo 1339958847 begin to download photo : 1339958851 download finish for photo 1339998272 begin to download photo : 1339958527 download finish for photo 1339958851 begin to download photo : 1339958960 download finish for photo 1339998274 download finish for photo 1339958316 begin to download photo : 1339958363 download finish for photo 1339958527 download finish for photo 1339958960 download finish for photo 1339958363 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901284_28-02-2025_01_50_46.pdf results_Auto_P20901284_28-02-2025_01_50_46.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901284_28-02-2025_01_50_46.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20901284','results_Auto_P20901284_28-02-2025_01_50_46.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901284_28-02-2025_01_50_46.pdf','pdf','','0.76','0.1199733468175425') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1339998274, 1339998272, 1339998237, 1339959122, 1339959044, 1339958960, 1339958851, 1339958849, 1339958847, 1339958697] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998274', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998272', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998237', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959122', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959044', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958960', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958851', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958849', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958847', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958697', None, None, None, None, None, '2621501') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.015983104705810547 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.121024131774902 time spend to save output : 0.01631784439086914 total time spend for step 10 : 6.1373419761657715 step11:split_time_score Fri Feb 28 01:50:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('18', 15),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 27022025 20901284 Nombre de photos uploadées : 15 / 23040 (0%) 27022025 20901284 Nombre de photos taguées (types de déchets): 0 / 15 (0%) 27022025 20901284 Nombre de photos taguées (volume) : 0 / 15 (0%) elapsed_time : load_data_split_time_score 1.6689300537109375e-06 elapsed_time : order_list_meta_photo_and_scores 5.4836273193359375e-06 ??????????????? elapsed_time : fill_and_build_computed_from_old_data 0.000774383544921875 elapsed_time : insert_dashboard_record_day_entry 0.033155202865600586 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901267 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901268 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901269 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901270 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901271 order by id desc limit 1 Qualite : 0.08224904095108558 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901272_28-02-2025_01_50_25.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901272 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901272 AND mptpi.`type`=4207 To do Qualite : 0.13080982673511443 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901273_28-02-2025_01_05_59.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901273 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901273 AND mptpi.`type`=3327 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901760 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901761 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901762 order by id desc limit 1 Qualite : 0.07982605740529508 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901763_28-02-2025_01_35_40.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901763 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11524 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11533 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11532 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11528 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11528 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11531 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11531 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11578 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11578 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11527 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11526 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11526 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11535 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11527 doesn't seem to be define in the database( WARNING : type of input 3 of step 11526 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11524 doesn't seem to be define in the database( WARNING : type of input 2 of step 11528 doesn't seem to be define in the database( WARNING : output 1 of step 11524 have datatype=2 whereas input 1 of step 11531 have datatype=7 WARNING : type of output 2 of step 11531 doesn't seem to be define in the database( WARNING : type of input 1 of step 11525 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11527 have datatype=10 whereas input 3 of step 11534 have datatype=6 WARNING : type of input 2 of step 11578 doesn't seem to be define in the database( WARNING : output 1 of step 11525 have datatype=7 whereas input 2 of step 11578 have datatype=None WARNING : type of output 3 of step 11578 doesn't seem to be define in the database( WARNING : type of input 1 of step 11527 doesn't seem to be define in the database( WARNING : output 0 of step 11527 have datatype=10 whereas input 0 of step 11584 have datatype=18 WARNING : type of input 5 of step 11534 doesn't seem to be define in the database( WARNING : output 0 of step 11584 have datatype=11 whereas input 5 of step 11534 have datatype=None WARNING : type of output 1 of step 11533 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : type of output 1 of step 11532 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : output 0 of step 11531 have datatype=1 whereas input 0 of step 11525 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901763 AND mptpi.`type`=4211 To do Qualite : 0.042362005557428556 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901764 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901764 AND mptpi.`type`=3327 To do Qualite : 0.19853583693421006 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901765_28-02-2025_01_40_01.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901765 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901765 AND mptpi.`type`=3327 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901276 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901766 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901278 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901279 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901767 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901768 order by id desc limit 1 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901769_28-02-2025_01_39_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901769 order by id desc limit 1 Qualite : 0.07069447068413548 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901282_28-02-2025_01_46_05.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901282 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901282 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901283 order by id desc limit 1 Qualite : 0.1199733468175425 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901284_28-02-2025_01_50_46.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901284 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901284 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901285 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901286 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901287 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901288 order by id desc limit 1 Qualite : 0.0662862777318847 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901301_28-02-2025_00_57_08.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901301 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901301 AND mptpi.`type`=4207 To do Qualite : 0.10326228661809093 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901314_28-02-2025_01_38_23.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901314 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901314 AND mptpi.`type`=4207 To do Qualite : 0.059403225739113 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901315_28-02-2025_01_08_40.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901315 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901315 AND mptpi.`type`=4207 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'27022025': {'nb_upload': 15, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1339998274, 1339998272, 1339998237, 1339959122, 1339959044, 1339958960, 1339958851, 1339958849, 1339958847, 1339958697] Looping around the photos to save general results len do output : 1 /20901284Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998274', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998272', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339998237', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959122', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339959044', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958960', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958851', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958849', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958847', None, None, None, None, None, '2621501') ('3459', None, None, None, None, None, None, None, '2621501') ('3459', '20901284', '1339958697', None, None, None, None, None, '2621501') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.017875194549560547 save_final save missing photos in datou_result : time spend for datou_step_exec : 17.017223358154297 time spend to save output : 0.018140792846679688 total time spend for step 11 : 17.035364151000977 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 151.07user 212.42system 6:44.06elapsed 89%CPU (0avgtext+0avgdata 4210088maxresident)k 780808inputs+131000outputs (685major+13928039minor)pagefaults 0swaps