python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2368861 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 44>10 please execute split_portfolio.py -i 20528860 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2584387'] with mtr_portfolio_ids : ['20528860'] and first list_photo_ids : [] new path : /proc/2368861/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.8023560047149658 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 14 00:34:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6022 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-14 00:34:32.413786: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-14 00:34:32.443162: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-14 00:34:32.444828: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fd324000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-14 00:34:32.444880: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-14 00:34:32.448700: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-14 00:34:32.703429: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3c689090 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-14 00:34:32.703476: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-14 00:34:32.704305: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-14 00:34:32.704623: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-14 00:34:32.706780: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-14 00:34:32.724525: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-14 00:34:32.725146: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-14 00:34:32.739840: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-14 00:34:32.742105: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-14 00:34:32.753622: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-14 00:34:32.754825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-14 00:34:32.754916: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-14 00:34:32.755574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-14 00:34:32.755590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-14 00:34:32.755599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-14 00:34:32.756697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5518 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-14 00:34:33.038766: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-14 00:34:33.038893: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-14 00:34:33.038927: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-14 00:34:33.038942: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-14 00:34:33.038956: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-14 00:34:33.038970: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-14 00:34:33.038985: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-14 00:34:33.039001: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-14 00:34:33.040038: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-14 00:34:33.041059: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-14 00:34:33.041093: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-14 00:34:33.041109: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-14 00:34:33.041123: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-14 00:34:33.041137: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-14 00:34:33.041159: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-14 00:34:33.041178: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-14 00:34:33.041199: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-14 00:34:33.042200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-14 00:34:33.042236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-14 00:34:33.042245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-14 00:34:33.042253: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-14 00:34:33.045665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5518 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-14 00:34:46.150975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-14 00:34:46.316466: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 13 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 17 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 17 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 17 Detection mask done ! Trying to reset tf kernel 2369286 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 733 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 6022 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0005352497100830078 nb_pixel_total : 17339 time to create 1 rle with old method : 0.01976776123046875 length of segment : 275 time for calcul the mask position with numpy : 0.0005955696105957031 nb_pixel_total : 26845 time to create 1 rle with old method : 0.03675127029418945 length of segment : 221 time for calcul the mask position with numpy : 0.00010633468627929688 nb_pixel_total : 2639 time to create 1 rle with old method : 0.004481077194213867 length of segment : 52 time for calcul the mask position with numpy : 0.0013866424560546875 nb_pixel_total : 68680 time to create 1 rle with old method : 0.07634925842285156 length of segment : 333 time for calcul the mask position with numpy : 0.001283884048461914 nb_pixel_total : 72096 time to create 1 rle with old method : 0.07526826858520508 length of segment : 428 time for calcul the mask position with numpy : 0.0004868507385253906 nb_pixel_total : 29164 time to create 1 rle with old method : 0.03526782989501953 length of segment : 239 time for calcul the mask position with numpy : 0.00027823448181152344 nb_pixel_total : 11047 time to create 1 rle with old method : 0.013039588928222656 length of segment : 125 time for calcul the mask position with numpy : 0.002767324447631836 nb_pixel_total : 110904 time to create 1 rle with old method : 0.12318253517150879 length of segment : 516 time for calcul the mask position with numpy : 0.0011823177337646484 nb_pixel_total : 28101 time to create 1 rle with old method : 0.031893014907836914 length of segment : 320 time for calcul the mask position with numpy : 0.006251811981201172 nb_pixel_total : 143707 time to create 1 rle with old method : 0.15748953819274902 length of segment : 768 time for calcul the mask position with numpy : 0.0003476142883300781 nb_pixel_total : 15267 time to create 1 rle with old method : 0.01703619956970215 length of segment : 198 time for calcul the mask position with numpy : 0.0017087459564208984 nb_pixel_total : 100541 time to create 1 rle with old method : 0.11348819732666016 length of segment : 511 time for calcul the mask position with numpy : 0.0036923885345458984 nb_pixel_total : 182565 time to create 1 rle with new method : 0.01558828353881836 length of segment : 747 time for calcul the mask position with numpy : 0.0003819465637207031 nb_pixel_total : 12830 time to create 1 rle with old method : 0.015117168426513672 length of segment : 160 time for calcul the mask position with numpy : 7.43865966796875e-05 nb_pixel_total : 1329 time to create 1 rle with old method : 0.0017428398132324219 length of segment : 55 time for calcul the mask position with numpy : 0.0005171298980712891 nb_pixel_total : 18012 time to create 1 rle with old method : 0.02168583869934082 length of segment : 215 time for calcul the mask position with numpy : 0.002038240432739258 nb_pixel_total : 103472 time to create 1 rle with old method : 0.13637733459472656 length of segment : 470 time for calcul the mask position with numpy : 0.0001888275146484375 nb_pixel_total : 6453 time to create 1 rle with old method : 0.0074384212493896484 length of segment : 112 time for calcul the mask position with numpy : 0.0039861202239990234 nb_pixel_total : 205728 time to create 1 rle with new method : 0.011310577392578125 length of segment : 621 time for calcul the mask position with numpy : 0.0004208087921142578 nb_pixel_total : 22736 time to create 1 rle with old method : 0.02674388885498047 length of segment : 188 time for calcul the mask position with numpy : 0.002218008041381836 nb_pixel_total : 125699 time to create 1 rle with old method : 0.1430344581604004 length of segment : 498 time for calcul the mask position with numpy : 0.00043463706970214844 nb_pixel_total : 17368 time to create 1 rle with old method : 0.019163846969604492 length of segment : 226 time for calcul the mask position with numpy : 0.0003325939178466797 nb_pixel_total : 17710 time to create 1 rle with old method : 0.019605398178100586 length of segment : 169 time for calcul the mask position with numpy : 0.0023391246795654297 nb_pixel_total : 94147 time to create 1 rle with old method : 0.1208195686340332 length of segment : 292 time for calcul the mask position with numpy : 8.96453857421875e-05 nb_pixel_total : 1812 time to create 1 rle with old method : 0.0022296905517578125 length of segment : 37 time for calcul the mask position with numpy : 0.00011444091796875 nb_pixel_total : 2367 time to create 1 rle with old method : 0.00296783447265625 length of segment : 63 time for calcul the mask position with numpy : 0.0003712177276611328 nb_pixel_total : 17287 time to create 1 rle with old method : 0.019544124603271484 length of segment : 214 time for calcul the mask position with numpy : 0.0028929710388183594 nb_pixel_total : 163334 time to create 1 rle with new method : 0.011849641799926758 length of segment : 464 time for calcul the mask position with numpy : 0.0010309219360351562 nb_pixel_total : 41843 time to create 1 rle with old method : 0.046919822692871094 length of segment : 248 time for calcul the mask position with numpy : 0.0015749931335449219 nb_pixel_total : 97837 time to create 1 rle with old method : 0.10878109931945801 length of segment : 488 time for calcul the mask position with numpy : 0.0010974407196044922 nb_pixel_total : 57338 time to create 1 rle with old method : 0.15463614463806152 length of segment : 344 time for calcul the mask position with numpy : 0.0002582073211669922 nb_pixel_total : 5425 time to create 1 rle with old method : 0.0065593719482421875 length of segment : 127 time for calcul the mask position with numpy : 0.0001666545867919922 nb_pixel_total : 3267 time to create 1 rle with old method : 0.00415492057800293 length of segment : 62 time for calcul the mask position with numpy : 0.0008144378662109375 nb_pixel_total : 45490 time to create 1 rle with old method : 0.05457639694213867 length of segment : 162 time for calcul the mask position with numpy : 0.00016951560974121094 nb_pixel_total : 4500 time to create 1 rle with old method : 0.005422115325927734 length of segment : 73 time for calcul the mask position with numpy : 0.007114410400390625 nb_pixel_total : 199293 time to create 1 rle with new method : 0.02337360382080078 length of segment : 552 time for calcul the mask position with numpy : 0.0028603076934814453 nb_pixel_total : 98661 time to create 1 rle with old method : 0.13097405433654785 length of segment : 330 time for calcul the mask position with numpy : 0.0024743080139160156 nb_pixel_total : 143188 time to create 1 rle with old method : 0.167557954788208 length of segment : 468 time for calcul the mask position with numpy : 0.0014462471008300781 nb_pixel_total : 97155 time to create 1 rle with old method : 0.1170969009399414 length of segment : 331 time for calcul the mask position with numpy : 0.0006954669952392578 nb_pixel_total : 11181 time to create 1 rle with old method : 0.014542579650878906 length of segment : 324 time for calcul the mask position with numpy : 0.0002148151397705078 nb_pixel_total : 8427 time to create 1 rle with old method : 0.014923095703125 length of segment : 96 time for calcul the mask position with numpy : 0.004240512847900391 nb_pixel_total : 197133 time to create 1 rle with new method : 0.009472846984863281 length of segment : 524 time for calcul the mask position with numpy : 0.00013756752014160156 nb_pixel_total : 5354 time to create 1 rle with old method : 0.006487131118774414 length of segment : 69 time for calcul the mask position with numpy : 6.985664367675781e-05 nb_pixel_total : 1555 time to create 1 rle with old method : 0.0023190975189208984 length of segment : 36 time for calcul the mask position with numpy : 0.002390623092651367 nb_pixel_total : 51339 time to create 1 rle with old method : 0.08353304862976074 length of segment : 371 time for calcul the mask position with numpy : 0.0018994808197021484 nb_pixel_total : 98242 time to create 1 rle with old method : 0.11083126068115234 length of segment : 533 time for calcul the mask position with numpy : 0.0015206336975097656 nb_pixel_total : 90354 time to create 1 rle with old method : 0.10426020622253418 length of segment : 477 time for calcul the mask position with numpy : 0.0025382041931152344 nb_pixel_total : 154314 time to create 1 rle with new method : 0.00590968132019043 length of segment : 419 time for calcul the mask position with numpy : 0.0033113956451416016 nb_pixel_total : 149043 time to create 1 rle with old method : 0.16594624519348145 length of segment : 475 time for calcul the mask position with numpy : 0.003253936767578125 nb_pixel_total : 115776 time to create 1 rle with old method : 0.13556957244873047 length of segment : 585 time for calcul the mask position with numpy : 0.000354766845703125 nb_pixel_total : 8207 time to create 1 rle with old method : 0.010053873062133789 length of segment : 112 time for calcul the mask position with numpy : 0.002580404281616211 nb_pixel_total : 132934 time to create 1 rle with old method : 0.16297411918640137 length of segment : 468 time for calcul the mask position with numpy : 0.010474681854248047 nb_pixel_total : 316852 time to create 1 rle with new method : 0.021281003952026367 length of segment : 1418 time for calcul the mask position with numpy : 0.0025365352630615234 nb_pixel_total : 81581 time to create 1 rle with old method : 0.08789515495300293 length of segment : 447 time for calcul the mask position with numpy : 0.0013041496276855469 nb_pixel_total : 63798 time to create 1 rle with old method : 0.09157896041870117 length of segment : 330 time for calcul the mask position with numpy : 0.0012629032135009766 nb_pixel_total : 33311 time to create 1 rle with old method : 0.0399324893951416 length of segment : 292 time for calcul the mask position with numpy : 0.0012345314025878906 nb_pixel_total : 36879 time to create 1 rle with old method : 0.0422215461730957 length of segment : 224 time for calcul the mask position with numpy : 0.0004699230194091797 nb_pixel_total : 11414 time to create 1 rle with old method : 0.019124507904052734 length of segment : 114 time for calcul the mask position with numpy : 0.002217531204223633 nb_pixel_total : 81548 time to create 1 rle with old method : 0.09002995491027832 length of segment : 432 time for calcul the mask position with numpy : 0.0003764629364013672 nb_pixel_total : 8585 time to create 1 rle with old method : 0.010594606399536133 length of segment : 109 time for calcul the mask position with numpy : 0.0005905628204345703 nb_pixel_total : 12320 time to create 1 rle with old method : 0.015010833740234375 length of segment : 169 time for calcul the mask position with numpy : 0.0005278587341308594 nb_pixel_total : 14298 time to create 1 rle with old method : 0.01733565330505371 length of segment : 147 time for calcul the mask position with numpy : 0.0016529560089111328 nb_pixel_total : 42863 time to create 1 rle with old method : 0.047510623931884766 length of segment : 335 time for calcul the mask position with numpy : 0.0013256072998046875 nb_pixel_total : 40991 time to create 1 rle with old method : 0.04501080513000488 length of segment : 243 time for calcul the mask position with numpy : 0.0032837390899658203 nb_pixel_total : 100770 time to create 1 rle with old method : 0.11453580856323242 length of segment : 329 time for calcul the mask position with numpy : 0.0001506805419921875 nb_pixel_total : 1911 time to create 1 rle with old method : 0.002325773239135742 length of segment : 76 time for calcul the mask position with numpy : 0.0017538070678710938 nb_pixel_total : 27674 time to create 1 rle with old method : 0.03228759765625 length of segment : 264 time for calcul the mask position with numpy : 0.00033473968505859375 nb_pixel_total : 4822 time to create 1 rle with old method : 0.008239984512329102 length of segment : 65 time for calcul the mask position with numpy : 0.00064849853515625 nb_pixel_total : 12226 time to create 1 rle with old method : 0.02294611930847168 length of segment : 138 time for calcul the mask position with numpy : 0.0017893314361572266 nb_pixel_total : 59053 time to create 1 rle with old method : 0.06765103340148926 length of segment : 285 time for calcul the mask position with numpy : 0.0013642311096191406 nb_pixel_total : 35236 time to create 1 rle with old method : 0.04268980026245117 length of segment : 293 time for calcul the mask position with numpy : 0.0013332366943359375 nb_pixel_total : 39109 time to create 1 rle with old method : 0.045185089111328125 length of segment : 347 time for calcul the mask position with numpy : 0.001127481460571289 nb_pixel_total : 26131 time to create 1 rle with old method : 0.03238630294799805 length of segment : 294 time for calcul the mask position with numpy : 0.002480030059814453 nb_pixel_total : 70354 time to create 1 rle with old method : 0.1071465015411377 length of segment : 370 time for calcul the mask position with numpy : 0.0006606578826904297 nb_pixel_total : 13854 time to create 1 rle with old method : 0.016810894012451172 length of segment : 259 time for calcul the mask position with numpy : 0.0003871917724609375 nb_pixel_total : 12988 time to create 1 rle with old method : 0.01561284065246582 length of segment : 109 time for calcul the mask position with numpy : 0.0008842945098876953 nb_pixel_total : 21813 time to create 1 rle with old method : 0.025792598724365234 length of segment : 342 time for calcul the mask position with numpy : 0.00024247169494628906 nb_pixel_total : 5054 time to create 1 rle with old method : 0.006166696548461914 length of segment : 91 time for calcul the mask position with numpy : 0.0028874874114990234 nb_pixel_total : 75487 time to create 1 rle with old method : 0.08528470993041992 length of segment : 393 time for calcul the mask position with numpy : 0.00018024444580078125 nb_pixel_total : 2172 time to create 1 rle with old method : 0.0028839111328125 length of segment : 43 time for calcul the mask position with numpy : 0.0033113956451416016 nb_pixel_total : 144652 time to create 1 rle with old method : 0.15887212753295898 length of segment : 695 time for calcul the mask position with numpy : 0.0021843910217285156 nb_pixel_total : 114697 time to create 1 rle with old method : 0.12746787071228027 length of segment : 541 time for calcul the mask position with numpy : 0.000331878662109375 nb_pixel_total : 10531 time to create 1 rle with old method : 0.012521743774414062 length of segment : 158 time for calcul the mask position with numpy : 0.0003178119659423828 nb_pixel_total : 9619 time to create 1 rle with old method : 0.01181173324584961 length of segment : 119 time for calcul the mask position with numpy : 0.0011973381042480469 nb_pixel_total : 68471 time to create 1 rle with old method : 0.0759880542755127 length of segment : 312 time for calcul the mask position with numpy : 0.004329681396484375 nb_pixel_total : 155123 time to create 1 rle with new method : 0.009238481521606445 length of segment : 584 time for calcul the mask position with numpy : 0.0002980232238769531 nb_pixel_total : 5742 time to create 1 rle with old method : 0.006857633590698242 length of segment : 106 time for calcul the mask position with numpy : 0.0004448890686035156 nb_pixel_total : 12564 time to create 1 rle with old method : 0.014730215072631836 length of segment : 134 time for calcul the mask position with numpy : 0.00186920166015625 nb_pixel_total : 55323 time to create 1 rle with old method : 0.06269288063049316 length of segment : 434 time for calcul the mask position with numpy : 0.0015671253204345703 nb_pixel_total : 36013 time to create 1 rle with old method : 0.040962934494018555 length of segment : 317 time for calcul the mask position with numpy : 0.0027937889099121094 nb_pixel_total : 102906 time to create 1 rle with old method : 0.11817717552185059 length of segment : 444 time for calcul the mask position with numpy : 0.0012712478637695312 nb_pixel_total : 37713 time to create 1 rle with old method : 0.043056488037109375 length of segment : 224 time for calcul the mask position with numpy : 0.0016956329345703125 nb_pixel_total : 95160 time to create 1 rle with old method : 0.1117701530456543 length of segment : 456 time for calcul the mask position with numpy : 0.0027761459350585938 nb_pixel_total : 106661 time to create 1 rle with old method : 0.11858415603637695 length of segment : 383 time for calcul the mask position with numpy : 0.0009398460388183594 nb_pixel_total : 32180 time to create 1 rle with old method : 0.037905216217041016 length of segment : 217 time for calcul the mask position with numpy : 0.0002357959747314453 nb_pixel_total : 3713 time to create 1 rle with old method : 0.004584550857543945 length of segment : 94 time for calcul the mask position with numpy : 0.002848386764526367 nb_pixel_total : 44952 time to create 1 rle with old method : 0.050863027572631836 length of segment : 444 time for calcul the mask position with numpy : 0.000179290771484375 nb_pixel_total : 1839 time to create 1 rle with old method : 0.0022864341735839844 length of segment : 66 time for calcul the mask position with numpy : 0.005109548568725586 nb_pixel_total : 213017 time to create 1 rle with new method : 0.013996124267578125 length of segment : 715 time for calcul the mask position with numpy : 0.0028476715087890625 nb_pixel_total : 101548 time to create 1 rle with old method : 0.10781264305114746 length of segment : 376 time for calcul the mask position with numpy : 0.0012531280517578125 nb_pixel_total : 41776 time to create 1 rle with old method : 0.04656624794006348 length of segment : 235 time for calcul the mask position with numpy : 0.05192899703979492 nb_pixel_total : 2117162 time to create 1 rle with new method : 0.9353764057159424 length of segment : 2907 time for calcul the mask position with numpy : 0.013797283172607422 nb_pixel_total : 865218 time to create 1 rle with new method : 0.032518863677978516 length of segment : 1827 time for calcul the mask position with numpy : 0.005491495132446289 nb_pixel_total : 437962 time to create 1 rle with new method : 0.015110492706298828 length of segment : 871 time for calcul the mask position with numpy : 0.0036437511444091797 nb_pixel_total : 316564 time to create 1 rle with new method : 0.007733821868896484 length of segment : 572 time for calcul the mask position with numpy : 0.0002067089080810547 nb_pixel_total : 8796 time to create 1 rle with old method : 0.010796546936035156 length of segment : 139 time for calcul the mask position with numpy : 0.006781578063964844 nb_pixel_total : 437862 time to create 1 rle with new method : 0.014665365219116211 length of segment : 618 time for calcul the mask position with numpy : 0.0020864009857177734 nb_pixel_total : 58646 time to create 1 rle with old method : 0.07395315170288086 length of segment : 295 time for calcul the mask position with numpy : 0.0006880760192871094 nb_pixel_total : 8604 time to create 1 rle with old method : 0.010816335678100586 length of segment : 243 time for calcul the mask position with numpy : 0.00015664100646972656 nb_pixel_total : 985 time to create 1 rle with old method : 0.0014719963073730469 length of segment : 41 time for calcul the mask position with numpy : 0.0012271404266357422 nb_pixel_total : 34682 time to create 1 rle with old method : 0.04163837432861328 length of segment : 433 time for calcul the mask position with numpy : 0.004579067230224609 nb_pixel_total : 151126 time to create 1 rle with new method : 0.010067462921142578 length of segment : 496 time for calcul the mask position with numpy : 0.0019440650939941406 nb_pixel_total : 19549 time to create 1 rle with old method : 0.03511786460876465 length of segment : 166 time for calcul the mask position with numpy : 0.0027375221252441406 nb_pixel_total : 75911 time to create 1 rle with old method : 0.11629056930541992 length of segment : 354 time for calcul the mask position with numpy : 0.0006797313690185547 nb_pixel_total : 25422 time to create 1 rle with old method : 0.03383779525756836 length of segment : 221 time for calcul the mask position with numpy : 0.0004074573516845703 nb_pixel_total : 7925 time to create 1 rle with old method : 0.010024547576904297 length of segment : 123 time for calcul the mask position with numpy : 0.0018739700317382812 nb_pixel_total : 79691 time to create 1 rle with old method : 0.11083579063415527 length of segment : 332 time for calcul the mask position with numpy : 0.0003376007080078125 nb_pixel_total : 10944 time to create 1 rle with old method : 0.014048099517822266 length of segment : 138 time for calcul the mask position with numpy : 0.00043773651123046875 nb_pixel_total : 13537 time to create 1 rle with old method : 0.01603388786315918 length of segment : 141 time for calcul the mask position with numpy : 0.0008282661437988281 nb_pixel_total : 33053 time to create 1 rle with old method : 0.0376889705657959 length of segment : 428 time for calcul the mask position with numpy : 0.0002617835998535156 nb_pixel_total : 3995 time to create 1 rle with old method : 0.006258487701416016 length of segment : 80 time for calcul the mask position with numpy : 0.002297639846801758 nb_pixel_total : 67293 time to create 1 rle with old method : 0.07715272903442383 length of segment : 441 time for calcul the mask position with numpy : 0.000408172607421875 nb_pixel_total : 15693 time to create 1 rle with old method : 0.019480228424072266 length of segment : 111 time for calcul the mask position with numpy : 0.0002999305725097656 nb_pixel_total : 9433 time to create 1 rle with old method : 0.011361360549926758 length of segment : 133 time for calcul the mask position with numpy : 0.00032401084899902344 nb_pixel_total : 6265 time to create 1 rle with old method : 0.007791280746459961 length of segment : 165 time for calcul the mask position with numpy : 0.0008523464202880859 nb_pixel_total : 30685 time to create 1 rle with old method : 0.036791086196899414 length of segment : 234 time for calcul the mask position with numpy : 0.002026081085205078 nb_pixel_total : 101739 time to create 1 rle with old method : 0.11869621276855469 length of segment : 385 time for calcul the mask position with numpy : 0.0002627372741699219 nb_pixel_total : 8275 time to create 1 rle with old method : 0.00976419448852539 length of segment : 119 time for calcul the mask position with numpy : 0.0003857612609863281 nb_pixel_total : 13970 time to create 1 rle with old method : 0.016296863555908203 length of segment : 148 time for calcul the mask position with numpy : 0.0001723766326904297 nb_pixel_total : 6242 time to create 1 rle with old method : 0.007815361022949219 length of segment : 78 time for calcul the mask position with numpy : 0.0002989768981933594 nb_pixel_total : 9643 time to create 1 rle with old method : 0.011894464492797852 length of segment : 138 time for calcul the mask position with numpy : 0.00035858154296875 nb_pixel_total : 13589 time to create 1 rle with old method : 0.015418052673339844 length of segment : 166 time for calcul the mask position with numpy : 0.001458883285522461 nb_pixel_total : 98882 time to create 1 rle with old method : 0.11020922660827637 length of segment : 366 time for calcul the mask position with numpy : 0.0022110939025878906 nb_pixel_total : 95832 time to create 1 rle with old method : 0.10923457145690918 length of segment : 469 time for calcul the mask position with numpy : 0.0016536712646484375 nb_pixel_total : 58451 time to create 1 rle with old method : 0.06828570365905762 length of segment : 389 time spent for convertir_results : 14.949766635894775 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 135 chid ids of type : 3663 Number RLEs to save : 44970 save missing photos in datou_result : time spend for datou_step_exec : 67.76603174209595 time spend to save output : 4.8169872760772705 total time spend for step 1 : 72.58301901817322 step2:crop_condition Fri Feb 14 00:35:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 135 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 14 About to insert : list_path_to_insert length 14 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 12 About to insert : list_path_to_insert length 12 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 41 About to insert : list_path_to_insert length 41 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1337462721, 1337330895, 1337330865, 1337329755, 1337329752, 1337329748, 1337329744, 1337329739, 1337329695, 1337329664] Looping around the photos to save general results len do output : 84 /-3674033546Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033566Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033556Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033569Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033600Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033593Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033610Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033618Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033632Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033653Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033647Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033654Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033666Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033663Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033549Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033544Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033541Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033578Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033592Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033591Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033595Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033598Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033606Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033631Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033646Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033659Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033577Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033585Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033624Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033629Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033552Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033543Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033547Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033562Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033563Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033568Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033558Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033576Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033570Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033573Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033586Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033581Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033588Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033604Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033602Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033596Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033608Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033594Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033603Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033609Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033621Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033612Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033613Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033616Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033620Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033611Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033636Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033633Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033640Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033638Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033628Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033630Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033644Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033643Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033651Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033656Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033652Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033658Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033650Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033674Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033661Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033540Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033564Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033584Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033582Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033615Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033657Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033672Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033668Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033548Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033625Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033673Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033553Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674033590Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337462721', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330895', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330865', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329755', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329752', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329748', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329744', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329739', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329695', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329664', None, None, None, None, None, '2584387') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 262 time used for this insertion : 0.08860373497009277 save_final save missing photos in datou_result : time spend for datou_step_exec : 38.83476996421814 time spend to save output : 0.09125757217407227 total time spend for step 2 : 38.92602753639221 step3:thcl Fri Feb 14 00:36:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.05968427658081055 time to convert the images to numpy array : 0.0384368896484375 time to import caffe and check if the image exist : 0.06416821479797363 time to convert the images to numpy array : 0.09712767601013184 time to import caffe and check if the image exist : 0.05952739715576172 time to convert the images to numpy array : 0.11376333236694336 time to import caffe and check if the image exist : 0.05917716026306152 time to convert the images to numpy array : 0.11589622497558594 time to import caffe and check if the image exist : 0.06398630142211914 time to convert the images to numpy array : 0.13454937934875488 time to import caffe and check if the image exist : 0.06189537048339844 time to convert the images to numpy array : 0.1406550407409668 time to import caffe and check if the image exist : 0.06351137161254883 time to convert the images to numpy array : 0.15104937553405762 time to import caffe and check if the image exist : 0.06815052032470703 time to convert the images to numpy array : 0.15128207206726074 time to import caffe and check if the image exist : 0.0650787353515625 time to convert the images to numpy array : 0.1889057159423828 time to import caffe and check if the image exist : 0.06696319580078125 time to convert the images to numpy array : 0.1993248462677002 total time to convert the images to numpy array : 0.4630281925201416 list photo_ids error: [] list photo_ids correct : [-3674033673, -3674033553, -3674033590, -3674033564, -3674033584, -3674033582, -3674033615, -3674033657, -3674033672, -3674033668, -3674033548, -3674033625, -3674033546, -3674033566, -3674033556, -3674033569, -3674033600, -3674033593, -3674033610, -3674033618, -3674033632, -3674033596, -3674033608, -3674033594, -3674033603, -3674033609, -3674033621, -3674033612, -3674033613, -3674033616, -3674033643, -3674033651, -3674033656, -3674033652, -3674033658, -3674033650, -3674033674, -3674033661, -3674033540, -3674033585, -3674033624, -3674033629, -3674033552, -3674033543, -3674033547, -3674033562, -3674033563, -3674033568, -3674033653, -3674033647, -3674033654, -3674033666, -3674033663, -3674033549, -3674033544, -3674033541, -3674033578, -3674033558, -3674033576, -3674033570, -3674033573, -3674033586, -3674033581, -3674033588, -3674033604, -3674033602, -3674033592, -3674033591, -3674033595, -3674033598, -3674033606, -3674033631, -3674033646, -3674033659, -3674033577, -3674033620, -3674033611, -3674033636, -3674033633, -3674033640, -3674033638, -3674033628, -3674033630, -3674033644] number of photos to traite : 84 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1141 wait 20 seconds l 3637 free memory gpu now : 1141 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 922 wait 20 seconds l 3637 free memory gpu now : 922 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.009627103805542 time used to do the prediction : 0.29505014419555664 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 920 wait 20 seconds l 3637 free memory gpu now : 920 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 920 wait 20 seconds l 3637 free memory gpu now : 920 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.0020146369934082 time used to do the prediction : 0.20426392555236816 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.288818359375e-05 save missing photos in datou_result : time spend for datou_step_exec : 89.77548718452454 time spend to save output : 0.0006647109985351562 total time spend for step 3 : 89.77615189552307 step4:merge_mask_thcl_custom Fri Feb 14 00:37:50 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 135 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1337462721, 1337330895, 1337330865, 1337329755, 1337329752, 1337329748, 1337329744, 1337329739, 1337329695, 1337329664] Looping around the photos to save general results len do output : 10 /1337462721Didn't retrieve data .Didn't retrieve data . /1337330895Didn't retrieve data .Didn't retrieve data . /1337330865Didn't retrieve data .Didn't retrieve data . /1337329755Didn't retrieve data .Didn't retrieve data . /1337329752Didn't retrieve data .Didn't retrieve data . /1337329748Didn't retrieve data .Didn't retrieve data . /1337329744Didn't retrieve data .Didn't retrieve data . /1337329739Didn't retrieve data .Didn't retrieve data . /1337329695Didn't retrieve data .Didn't retrieve data . /1337329664Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337462721', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330895', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330865', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329755', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329752', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329748', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329744', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329739', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329695', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329664', None, None, None, None, None, '2584387') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.015391111373901367 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.07935690879821777 time spend to save output : 0.01594829559326172 total time spend for step 4 : 0.09530520439147949 step5:rle_unique_nms_with_priority Fri Feb 14 00:37:50 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 14 nb_hashtags : 6 time to prepare the origin masks : 5.610039234161377 time for calcul the mask position with numpy : 0.24166059494018555 nb_pixel_total : 6363518 time to create 1 rle with new method : 0.44586706161499023 time for calcul the mask position with numpy : 0.02043008804321289 nb_pixel_total : 26807 time to create 1 rle with old method : 0.032698631286621094 time for calcul the mask position with numpy : 0.04034733772277832 nb_pixel_total : 10991 time to create 1 rle with old method : 0.011559247970581055 time for calcul the mask position with numpy : 0.032474517822265625 nb_pixel_total : 17348 time to create 1 rle with old method : 0.020685434341430664 time for calcul the mask position with numpy : 0.036324262619018555 nb_pixel_total : 1316 time to create 1 rle with old method : 0.0015149116516113281 time for calcul the mask position with numpy : 0.03480052947998047 nb_pixel_total : 28096 time to create 1 rle with old method : 0.031472206115722656 time for calcul the mask position with numpy : 0.03422689437866211 nb_pixel_total : 15229 time to create 1 rle with old method : 0.01759934425354004 time for calcul the mask position with numpy : 0.033829689025878906 nb_pixel_total : 12794 time to create 1 rle with old method : 0.013828754425048828 time for calcul the mask position with numpy : 0.033599138259887695 nb_pixel_total : 72020 time to create 1 rle with old method : 0.07769060134887695 time for calcul the mask position with numpy : 0.03879046440124512 nb_pixel_total : 29121 time to create 1 rle with old method : 0.033196449279785156 time for calcul the mask position with numpy : 0.03433990478515625 nb_pixel_total : 2628 time to create 1 rle with old method : 0.0031223297119140625 time for calcul the mask position with numpy : 0.03586077690124512 nb_pixel_total : 102246 time to create 1 rle with old method : 0.12065005302429199 time for calcul the mask position with numpy : 0.04113054275512695 nb_pixel_total : 110658 time to create 1 rle with old method : 0.1391596794128418 time for calcul the mask position with numpy : 0.022725343704223633 nb_pixel_total : 188988 time to create 1 rle with new method : 0.3350505828857422 time for calcul the mask position with numpy : 0.03224658966064453 nb_pixel_total : 68480 time to create 1 rle with old method : 0.07323431968688965 create new chi : 2.128312826156616 time to delete rle : 0.011547088623046875 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 10603 TO DO : save crop sub photo not yet done ! save time : 1.448659896850586 nb_obj : 13 nb_hashtags : 5 time to prepare the origin masks : 6.60190749168396 time for calcul the mask position with numpy : 0.25284743309020996 nb_pixel_total : 6246584 time to create 1 rle with new method : 0.3057894706726074 time for calcul the mask position with numpy : 0.021455049514770508 nb_pixel_total : 41710 time to create 1 rle with old method : 0.04554247856140137 time for calcul the mask position with numpy : 0.023595809936523438 nb_pixel_total : 1795 time to create 1 rle with old method : 0.002134084701538086 time for calcul the mask position with numpy : 0.023824214935302734 nb_pixel_total : 163115 time to create 1 rle with new method : 0.30038976669311523 time for calcul the mask position with numpy : 0.023386716842651367 nb_pixel_total : 2321 time to create 1 rle with old method : 0.0027437210083007812 time for calcul the mask position with numpy : 0.03391146659851074 nb_pixel_total : 17329 time to create 1 rle with old method : 0.020090818405151367 time for calcul the mask position with numpy : 0.03339862823486328 nb_pixel_total : 17261 time to create 1 rle with old method : 0.01973271369934082 time for calcul the mask position with numpy : 0.03526568412780762 nb_pixel_total : 103081 time to create 1 rle with old method : 0.1161811351776123 time for calcul the mask position with numpy : 0.034096717834472656 nb_pixel_total : 124922 time to create 1 rle with old method : 0.13807463645935059 time for calcul the mask position with numpy : 0.035396575927734375 nb_pixel_total : 191498 time to create 1 rle with new method : 0.32654500007629395 time for calcul the mask position with numpy : 0.036031246185302734 nb_pixel_total : 22660 time to create 1 rle with old method : 0.02597498893737793 time for calcul the mask position with numpy : 0.03466439247131348 nb_pixel_total : 93836 time to create 1 rle with old method : 0.1030569076538086 time for calcul the mask position with numpy : 0.02030014991760254 nb_pixel_total : 6442 time to create 1 rle with old method : 0.007154941558837891 time for calcul the mask position with numpy : 0.025645971298217773 nb_pixel_total : 17686 time to create 1 rle with old method : 0.020092248916625977 create new chi : 2.151365280151367 time to delete rle : 0.0009326934814453125 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 9265 TO DO : save crop sub photo not yet done ! save time : 1.2527952194213867 nb_obj : 11 nb_hashtags : 3 time to prepare the origin masks : 7.351677417755127 time for calcul the mask position with numpy : 0.5175068378448486 nb_pixel_total : 6386065 time to create 1 rle with new method : 0.3361999988555908 time for calcul the mask position with numpy : 0.023358821868896484 nb_pixel_total : 2591 time to create 1 rle with old method : 0.004747867584228516 time for calcul the mask position with numpy : 0.02204132080078125 nb_pixel_total : 11052 time to create 1 rle with old method : 0.013146638870239258 time for calcul the mask position with numpy : 0.022240638732910156 nb_pixel_total : 143001 time to create 1 rle with old method : 0.156754732131958 time for calcul the mask position with numpy : 0.02112555503845215 nb_pixel_total : 45164 time to create 1 rle with old method : 0.05074143409729004 time for calcul the mask position with numpy : 0.02491140365600586 nb_pixel_total : 197323 time to create 1 rle with new method : 0.6345996856689453 time for calcul the mask position with numpy : 0.02601003646850586 nb_pixel_total : 57380 time to create 1 rle with old method : 0.06406641006469727 time for calcul the mask position with numpy : 0.022967100143432617 nb_pixel_total : 96861 time to create 1 rle with old method : 0.11912035942077637 time for calcul the mask position with numpy : 0.022472620010375977 nb_pixel_total : 97699 time to create 1 rle with old method : 0.10783553123474121 time for calcul the mask position with numpy : 0.02036595344543457 nb_pixel_total : 4450 time to create 1 rle with old method : 0.004978656768798828 time for calcul the mask position with numpy : 0.02056264877319336 nb_pixel_total : 3252 time to create 1 rle with old method : 0.0034618377685546875 time for calcul the mask position with numpy : 0.01995992660522461 nb_pixel_total : 5402 time to create 1 rle with old method : 0.005937814712524414 create new chi : 2.3312854766845703 time to delete rle : 0.0007598400115966797 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 8364 TO DO : save crop sub photo not yet done ! save time : 1.5104892253875732 nb_obj : 13 nb_hashtags : 3 time to prepare the origin masks : 7.9040656089782715 time for calcul the mask position with numpy : 0.38751912117004395 nb_pixel_total : 5997234 time to create 1 rle with new method : 0.41077232360839844 time for calcul the mask position with numpy : 0.020975589752197266 nb_pixel_total : 5124 time to create 1 rle with old method : 0.006042003631591797 time for calcul the mask position with numpy : 0.021771907806396484 nb_pixel_total : 5023 time to create 1 rle with old method : 0.0056362152099609375 time for calcul the mask position with numpy : 0.02477407455444336 nb_pixel_total : 115364 time to create 1 rle with old method : 0.12716960906982422 time for calcul the mask position with numpy : 0.038109540939331055 nb_pixel_total : 6642 time to create 1 rle with old method : 0.008271217346191406 time for calcul the mask position with numpy : 0.03432345390319824 nb_pixel_total : 19781 time to create 1 rle with old method : 0.022526979446411133 time for calcul the mask position with numpy : 0.038178443908691406 nb_pixel_total : 41149 time to create 1 rle with old method : 0.04760313034057617 time for calcul the mask position with numpy : 0.03407406806945801 nb_pixel_total : 132478 time to create 1 rle with old method : 0.14335083961486816 time for calcul the mask position with numpy : 0.033969879150390625 nb_pixel_total : 153879 time to create 1 rle with new method : 0.5397868156433105 time for calcul the mask position with numpy : 0.036440134048461914 nb_pixel_total : 1547 time to create 1 rle with old method : 0.0018949508666992188 time for calcul the mask position with numpy : 0.03714704513549805 nb_pixel_total : 50982 time to create 1 rle with old method : 0.060693979263305664 time for calcul the mask position with numpy : 0.02741217613220215 nb_pixel_total : 316068 time to create 1 rle with new method : 0.6042635440826416 time for calcul the mask position with numpy : 0.02180027961730957 nb_pixel_total : 8370 time to create 1 rle with old method : 0.009247779846191406 time for calcul the mask position with numpy : 0.022989273071289062 nb_pixel_total : 196599 time to create 1 rle with new method : 0.48361945152282715 create new chi : 3.3712267875671387 time to delete rle : 0.0017399787902832031 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 12351 TO DO : save crop sub photo not yet done ! save time : 1.2963061332702637 nb_obj : 16 nb_hashtags : 5 time to prepare the origin masks : 9.413369178771973 time for calcul the mask position with numpy : 0.4194183349609375 nb_pixel_total : 6562035 time to create 1 rle with new method : 0.6042201519012451 time for calcul the mask position with numpy : 0.03379988670349121 nb_pixel_total : 63536 time to create 1 rle with old method : 0.06946921348571777 time for calcul the mask position with numpy : 0.035727739334106445 nb_pixel_total : 979 time to create 1 rle with old method : 0.0011827945709228516 time for calcul the mask position with numpy : 0.03372621536254883 nb_pixel_total : 42439 time to create 1 rle with old method : 0.04701685905456543 time for calcul the mask position with numpy : 0.03712654113769531 nb_pixel_total : 40889 time to create 1 rle with old method : 0.044547319412231445 time for calcul the mask position with numpy : 0.0338902473449707 nb_pixel_total : 100677 time to create 1 rle with old method : 0.1031496524810791 time for calcul the mask position with numpy : 0.03249096870422363 nb_pixel_total : 36856 time to create 1 rle with old method : 0.03893232345581055 time for calcul the mask position with numpy : 0.03936505317687988 nb_pixel_total : 81476 time to create 1 rle with old method : 0.0891268253326416 time for calcul the mask position with numpy : 0.03365516662597656 nb_pixel_total : 859 time to create 1 rle with old method : 0.0011508464813232422 time for calcul the mask position with numpy : 0.02961134910583496 nb_pixel_total : 27288 time to create 1 rle with old method : 0.030221223831176758 time for calcul the mask position with numpy : 0.032376766204833984 nb_pixel_total : 1902 time to create 1 rle with old method : 0.0021162033081054688 time for calcul the mask position with numpy : 0.03243112564086914 nb_pixel_total : 32870 time to create 1 rle with old method : 0.03963422775268555 time for calcul the mask position with numpy : 0.036568403244018555 nb_pixel_total : 14160 time to create 1 rle with old method : 0.01483464241027832 time for calcul the mask position with numpy : 0.03246617317199707 nb_pixel_total : 8474 time to create 1 rle with old method : 0.009588241577148438 time for calcul the mask position with numpy : 0.032375335693359375 nb_pixel_total : 12315 time to create 1 rle with old method : 0.013591766357421875 time for calcul the mask position with numpy : 0.033003807067871094 nb_pixel_total : 12147 time to create 1 rle with old method : 0.013558149337768555 time for calcul the mask position with numpy : 0.04013180732727051 nb_pixel_total : 11338 time to create 1 rle with old method : 0.013319253921508789 create new chi : 2.144796133041382 time to delete rle : 0.0009245872497558594 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 8664 TO DO : save crop sub photo not yet done ! save time : 1.3125224113464355 nb_obj : 17 nb_hashtags : 6 time to prepare the origin masks : 10.697476625442505 time for calcul the mask position with numpy : 0.2607271671295166 nb_pixel_total : 6326088 time to create 1 rle with new method : 0.28560328483581543 time for calcul the mask position with numpy : 0.02068614959716797 nb_pixel_total : 145202 time to create 1 rle with old method : 0.15356850624084473 time for calcul the mask position with numpy : 0.01969766616821289 nb_pixel_total : 22370 time to create 1 rle with old method : 0.024058103561401367 time for calcul the mask position with numpy : 0.022910356521606445 nb_pixel_total : 35275 time to create 1 rle with old method : 0.04032278060913086 time for calcul the mask position with numpy : 0.025426387786865234 nb_pixel_total : 4494 time to create 1 rle with old method : 0.005488157272338867 time for calcul the mask position with numpy : 0.02397632598876953 nb_pixel_total : 68372 time to create 1 rle with old method : 0.0748128890991211 time for calcul the mask position with numpy : 0.022066116333007812 nb_pixel_total : 12903 time to create 1 rle with old method : 0.014876365661621094 time for calcul the mask position with numpy : 0.024764299392700195 nb_pixel_total : 9569 time to create 1 rle with old method : 0.010692834854125977 time for calcul the mask position with numpy : 0.02218484878540039 nb_pixel_total : 10485 time to create 1 rle with old method : 0.012244462966918945 time for calcul the mask position with numpy : 0.02148151397705078 nb_pixel_total : 2148 time to create 1 rle with old method : 0.002510547637939453 time for calcul the mask position with numpy : 0.021055936813354492 nb_pixel_total : 5045 time to create 1 rle with old method : 0.006021022796630859 time for calcul the mask position with numpy : 0.021044492721557617 nb_pixel_total : 13771 time to create 1 rle with old method : 0.015557050704956055 time for calcul the mask position with numpy : 0.0208127498626709 nb_pixel_total : 40163 time to create 1 rle with old method : 0.04460406303405762 time for calcul the mask position with numpy : 0.02020716667175293 nb_pixel_total : 39567 time to create 1 rle with old method : 0.04431796073913574 time for calcul the mask position with numpy : 0.020900249481201172 nb_pixel_total : 75242 time to create 1 rle with old method : 0.08105874061584473 time for calcul the mask position with numpy : 0.02353358268737793 nb_pixel_total : 25630 time to create 1 rle with old method : 0.029438018798828125 time for calcul the mask position with numpy : 0.0211789608001709 nb_pixel_total : 154983 time to create 1 rle with new method : 0.4212815761566162 time for calcul the mask position with numpy : 0.02098369598388672 nb_pixel_total : 58933 time to create 1 rle with old method : 0.06405377388000488 create new chi : 2.0230014324188232 time to delete rle : 0.0010018348693847656 batch 1 Loaded 18 chid ids of type : 3726 Number RLEs to save : 11527 TO DO : save crop sub photo not yet done ! save time : 1.3294594287872314 nb_obj : 14 nb_hashtags : 5 time to prepare the origin masks : 9.973470449447632 time for calcul the mask position with numpy : 0.6200535297393799 nb_pixel_total : 6415006 time to create 1 rle with new method : 0.6344757080078125 time for calcul the mask position with numpy : 0.022566795349121094 nb_pixel_total : 55025 time to create 1 rle with old method : 0.06186032295227051 time for calcul the mask position with numpy : 0.02179408073425293 nb_pixel_total : 8720 time to create 1 rle with old method : 0.0102386474609375 time for calcul the mask position with numpy : 0.022324562072753906 nb_pixel_total : 85797 time to create 1 rle with old method : 0.09214520454406738 time for calcul the mask position with numpy : 0.02064800262451172 nb_pixel_total : 35928 time to create 1 rle with old method : 0.03847694396972656 time for calcul the mask position with numpy : 0.02029728889465332 nb_pixel_total : 1834 time to create 1 rle with old method : 0.0020530223846435547 time for calcul the mask position with numpy : 0.020883798599243164 nb_pixel_total : 3694 time to create 1 rle with old method : 0.0041162967681884766 time for calcul the mask position with numpy : 0.02165055274963379 nb_pixel_total : 94686 time to create 1 rle with old method : 0.0996546745300293 time for calcul the mask position with numpy : 0.021269559860229492 nb_pixel_total : 32036 time to create 1 rle with old method : 0.03798532485961914 time for calcul the mask position with numpy : 0.02196192741394043 nb_pixel_total : 212690 time to create 1 rle with new method : 0.8340818881988525 time for calcul the mask position with numpy : 0.022561073303222656 nb_pixel_total : 109 time to create 1 rle with old method : 0.0003261566162109375 time for calcul the mask position with numpy : 0.03137350082397461 nb_pixel_total : 12511 time to create 1 rle with old method : 0.014485836029052734 time for calcul the mask position with numpy : 0.032666921615600586 nb_pixel_total : 41672 time to create 1 rle with old method : 0.06915497779846191 time for calcul the mask position with numpy : 0.03717470169067383 nb_pixel_total : 5727 time to create 1 rle with old method : 0.006434440612792969 time for calcul the mask position with numpy : 0.03152751922607422 nb_pixel_total : 44805 time to create 1 rle with old method : 0.06051015853881836 create new chi : 3.0074706077575684 time to delete rle : 0.0010004043579101562 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 10018 TO DO : save crop sub photo not yet done ! save time : 1.1321933269500732 nb_obj : 6 nb_hashtags : 4 time to prepare the origin masks : 8.539746284484863 time for calcul the mask position with numpy : 0.5609879493713379 nb_pixel_total : 3566733 time to create 1 rle with new method : 1.6814510822296143 time for calcul the mask position with numpy : 0.021066665649414062 nb_pixel_total : 134916 time to create 1 rle with old method : 0.14396357536315918 time for calcul the mask position with numpy : 0.020541667938232422 nb_pixel_total : 7802 time to create 1 rle with old method : 0.008925199508666992 time for calcul the mask position with numpy : 0.02375316619873047 nb_pixel_total : 468144 time to create 1 rle with new method : 1.2008750438690186 time for calcul the mask position with numpy : 0.024435758590698242 nb_pixel_total : 437783 time to create 1 rle with new method : 0.4468977451324463 time for calcul the mask position with numpy : 0.04038190841674805 nb_pixel_total : 2118975 time to create 1 rle with new method : 1.2939298152923584 time for calcul the mask position with numpy : 0.024296045303344727 nb_pixel_total : 315887 time to create 1 rle with new method : 0.6342964172363281 create new chi : 6.265699625015259 time to delete rle : 0.0012912750244140625 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 13793 TO DO : save crop sub photo not yet done ! save time : 2.2484638690948486 nb_obj : 13 nb_hashtags : 6 time to prepare the origin masks : 18.062121868133545 time for calcul the mask position with numpy : 2.815105438232422 nb_pixel_total : 6604217 time to create 1 rle with new method : 1.038069248199463 time for calcul the mask position with numpy : 0.03117203712463379 nb_pixel_total : 151438 time to create 1 rle with new method : 1.8427350521087646 time for calcul the mask position with numpy : 0.03212285041809082 nb_pixel_total : 33144 time to create 1 rle with old method : 0.03479146957397461 time for calcul the mask position with numpy : 0.03445863723754883 nb_pixel_total : 34502 time to create 1 rle with old method : 0.036263465881347656 time for calcul the mask position with numpy : 0.03144574165344238 nb_pixel_total : 25328 time to create 1 rle with old method : 0.025970458984375 time for calcul the mask position with numpy : 0.031162738800048828 nb_pixel_total : 19445 time to create 1 rle with old method : 0.020440101623535156 time for calcul the mask position with numpy : 0.03243899345397949 nb_pixel_total : 10888 time to create 1 rle with old method : 0.011518716812133789 time for calcul the mask position with numpy : 0.029970169067382812 nb_pixel_total : 7866 time to create 1 rle with old method : 0.008436203002929688 time for calcul the mask position with numpy : 0.019242525100708008 nb_pixel_total : 2413 time to create 1 rle with old method : 0.003056049346923828 time for calcul the mask position with numpy : 0.01969313621520996 nb_pixel_total : 984 time to create 1 rle with old method : 0.0011029243469238281 time for calcul the mask position with numpy : 0.02123093605041504 nb_pixel_total : 79500 time to create 1 rle with old method : 0.08703112602233887 time for calcul the mask position with numpy : 0.022826433181762695 nb_pixel_total : 58469 time to create 1 rle with old method : 0.06410813331604004 time for calcul the mask position with numpy : 0.021733999252319336 nb_pixel_total : 8559 time to create 1 rle with old method : 0.009626626968383789 time for calcul the mask position with numpy : 0.027762174606323242 nb_pixel_total : 13487 time to create 1 rle with old method : 0.01436305046081543 create new chi : 6.430320978164673 time to delete rle : 0.0009007453918457031 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 8432 TO DO : save crop sub photo not yet done ! save time : 1.1064965724945068 nb_obj : 14 nb_hashtags : 3 time to prepare the origin masks : 16.09224820137024 time for calcul the mask position with numpy : 0.7042965888977051 nb_pixel_total : 6613732 time to create 1 rle with new method : 1.3427684307098389 time for calcul the mask position with numpy : 0.03403663635253906 nb_pixel_total : 67001 time to create 1 rle with old method : 0.0700221061706543 time for calcul the mask position with numpy : 0.024484634399414062 nb_pixel_total : 58288 time to create 1 rle with old method : 0.0608217716217041 time for calcul the mask position with numpy : 0.01940751075744629 nb_pixel_total : 831 time to create 1 rle with old method : 0.0011568069458007812 time for calcul the mask position with numpy : 0.020375490188598633 nb_pixel_total : 101503 time to create 1 rle with old method : 0.1060037612915039 time for calcul the mask position with numpy : 0.02006983757019043 nb_pixel_total : 13584 time to create 1 rle with old method : 0.01466059684753418 time for calcul the mask position with numpy : 0.020729780197143555 nb_pixel_total : 9578 time to create 1 rle with old method : 0.010754108428955078 time for calcul the mask position with numpy : 0.019961833953857422 nb_pixel_total : 6232 time to create 1 rle with old method : 0.006388425827026367 time for calcul the mask position with numpy : 0.020190000534057617 nb_pixel_total : 13922 time to create 1 rle with old method : 0.014658689498901367 time for calcul the mask position with numpy : 0.019646406173706055 nb_pixel_total : 9410 time to create 1 rle with old method : 0.010242462158203125 time for calcul the mask position with numpy : 0.020335674285888672 nb_pixel_total : 8180 time to create 1 rle with old method : 0.008772850036621094 time for calcul the mask position with numpy : 0.020290374755859375 nb_pixel_total : 30605 time to create 1 rle with old method : 0.03165459632873535 time for calcul the mask position with numpy : 0.01907038688659668 nb_pixel_total : 6233 time to create 1 rle with old method : 0.006569862365722656 time for calcul the mask position with numpy : 0.01950216293334961 nb_pixel_total : 15537 time to create 1 rle with old method : 0.016872882843017578 time for calcul the mask position with numpy : 0.02032327651977539 nb_pixel_total : 95604 time to create 1 rle with old method : 0.10009217262268066 create new chi : 2.84509015083313 time to delete rle : 0.0007586479187011719 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 8298 TO DO : save crop sub photo not yet done ! save time : 0.8889765739440918 map_output_result : {1337462721: (0.0, 'Should be the crop_list due to order', 0.0), 1337330895: (0.0, 'Should be the crop_list due to order', 0.0), 1337330865: (0.0, 'Should be the crop_list due to order', 0.0), 1337329755: (0.0, 'Should be the crop_list due to order', 0.0), 1337329752: (0.0, 'Should be the crop_list due to order', 0.0), 1337329748: (0.0, 'Should be the crop_list due to order', 0.0), 1337329744: (0.0, 'Should be the crop_list due to order', 0.0), 1337329739: (0.0, 'Should be the crop_list due to order', 0.0), 1337329695: (0.0, 'Should be the crop_list due to order', 0.0), 1337329664: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1337462721, 1337330895, 1337330865, 1337329755, 1337329752, 1337329748, 1337329744, 1337329739, 1337329695, 1337329664] Looping around the photos to save general results len do output : 10 /1337462721.Didn't retrieve data . /1337330895.Didn't retrieve data . /1337330865.Didn't retrieve data . /1337329755.Didn't retrieve data . /1337329752.Didn't retrieve data . /1337329748.Didn't retrieve data . /1337329744.Didn't retrieve data . /1337329739.Didn't retrieve data . /1337329695.Didn't retrieve data . /1337329664.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337462721', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330895', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330865', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329755', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329752', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329748', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329744', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329739', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329695', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329664', None, None, None, None, None, '2584387') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01590871810913086 save_final save missing photos in datou_result : time spend for datou_step_exec : 147.49164199829102 time spend to save output : 0.0164034366607666 total time spend for step 5 : 147.50804543495178 step6:crop_condition Fri Feb 14 00:40:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 141 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739490020_2368861 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.1251118183135986 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739490023_2368861 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6173684597015381 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739490027_2368861 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.5064339637756348 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 4869462 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739490035_2368861 we have uploaded 9 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.462599515914917 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739490040_2368861 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.3101089000701904 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1337462721, 1337330895, 1337330865, 1337329755, 1337329752, 1337329748, 1337329744, 1337329739, 1337329695, 1337329664] Looping around the photos to save general results len do output : 23 /1337471020Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471021Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471022Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471024Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471027Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471028Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471029Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471030Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471031Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471032Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471035Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471036Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471037Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471038Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471039Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471040Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471041Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471043Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471044Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471045Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471046Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471047Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337471048Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337462721', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330895', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330865', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329755', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329752', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329748', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329744', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329739', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329695', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329664', None, None, None, None, None, '2584387') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 79 time used for this insertion : 0.039464473724365234 save_final save missing photos in datou_result : time spend for datou_step_exec : 25.346295595169067 time spend to save output : 0.04024863243103027 total time spend for step 6 : 25.386544227600098 step7:ventilate_hashtags_in_portfolio Fri Feb 14 00:40:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20528860 get user id for portfolio 20528860 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20528860 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_gris','kraft','papier','Carton_brun','Teint_Dans_La_Masse','metal','cartonnette','plastique','autre_refus','mal_croppe','flou','environnement')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20528860 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_gris','kraft','papier','Carton_brun','Teint_Dans_La_Masse','metal','cartonnette','plastique','autre_refus','mal_croppe','flou','environnement')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20529403,20529404,20529405,20529406,20529407,20529408,20529409,20529410,20529411,20529412,20529413,20529414?tags=mal_croppe,Carton_gris,plastique,kraft,Teint_Dans_La_Masse,flou,Carton_brun,papier,environnement,autre_refus,metal,cartonnette Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1337462721, 1337330895, 1337330865, 1337329755, 1337329752, 1337329748, 1337329744, 1337329739, 1337329695, 1337329664] Looping around the photos to save general results len do output : 1 /20528860. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337462721', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330895', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330865', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329755', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329752', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329748', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329744', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329739', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329695', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329664', None, None, None, None, None, '2584387') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.02019977569580078 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.44146180152893066 time spend to save output : 0.020473718643188477 total time spend for step 7 : 0.46193552017211914 step8:final Fri Feb 14 00:40:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1337462721: ('0.062116323844456485',), 1337330895: ('0.062116323844456485',), 1337330865: ('0.062116323844456485',), 1337329755: ('0.062116323844456485',), 1337329752: ('0.062116323844456485',), 1337329748: ('0.062116323844456485',), 1337329744: ('0.062116323844456485',), 1337329739: ('0.062116323844456485',), 1337329695: ('0.062116323844456485',), 1337329664: ('0.062116323844456485',)} new output for save of step final : {1337462721: ('0.062116323844456485',), 1337330895: ('0.062116323844456485',), 1337330865: ('0.062116323844456485',), 1337329755: ('0.062116323844456485',), 1337329752: ('0.062116323844456485',), 1337329748: ('0.062116323844456485',), 1337329744: ('0.062116323844456485',), 1337329739: ('0.062116323844456485',), 1337329695: ('0.062116323844456485',), 1337329664: ('0.062116323844456485',)} [1337462721, 1337330895, 1337330865, 1337329755, 1337329752, 1337329748, 1337329744, 1337329739, 1337329695, 1337329664] Looping around the photos to save general results len do output : 10 /1337462721.Didn't retrieve data . /1337330895.Didn't retrieve data . /1337330865.Didn't retrieve data . /1337329755.Didn't retrieve data . /1337329752.Didn't retrieve data . /1337329748.Didn't retrieve data . /1337329744.Didn't retrieve data . /1337329739.Didn't retrieve data . /1337329695.Didn't retrieve data . /1337329664.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337462721', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330895', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330865', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329755', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329752', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329748', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329744', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329739', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329695', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329664', None, None, None, None, None, '2584387') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.015113592147827148 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10760617256164551 time spend to save output : 0.015493154525756836 total time spend for step 8 : 0.12309932708740234 step9:velours_tree Fri Feb 14 00:40:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.04390239715576172 time spend to save output : 3.361701965332031e-05 total time spend for step 9 : 0.04393601417541504 step10:send_mail_cod Fri Feb 14 00:40:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20528860_14-02-2025_00_40_44.pdf 20529403 imagette205294031739490044 20529405 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette205294051739490044 20529406 imagette205294061739490044 20529408 imagette205294081739490044 20529412 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette205294121739490044 20529413 imagette205294131739490045 20529414 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette205294141739490045 20529404 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette205294041739490045 20529407 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette205294071739490046 20529409 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette205294091739490047 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20528860 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20529403,20529404,20529405,20529406,20529407,20529408,20529409,20529410,20529411,20529412,20529413,20529414?tags=mal_croppe,Carton_gris,plastique,kraft,Teint_Dans_La_Masse,flou,Carton_brun,papier,environnement,autre_refus,metal,cartonnette your option no_mail is active, we will not send the real mail to your client args[1337462721] : ('0.062116323844456485',) no score found for photo 1337462721 apple ('0.062116323844456485',) We are sending mail with results at report@fotonower.com args[1337330895] : ('0.062116323844456485',) no score found for photo 1337330895 apple ('0.062116323844456485',) We are sending mail with results at report@fotonower.com args[1337330865] : ('0.062116323844456485',) no score found for photo 1337330865 apple ('0.062116323844456485',) We are sending mail with results at report@fotonower.com args[1337329755] : ('0.062116323844456485',) no score found for photo 1337329755 apple ('0.062116323844456485',) We are sending mail with results at report@fotonower.com args[1337329752] : ('0.062116323844456485',) no score found for photo 1337329752 apple ('0.062116323844456485',) We are sending mail with results at report@fotonower.com args[1337329748] : ('0.062116323844456485',) no score found for photo 1337329748 apple ('0.062116323844456485',) We are sending mail with results at report@fotonower.com args[1337329744] : ('0.062116323844456485',) no score found for photo 1337329744 apple ('0.062116323844456485',) We are sending mail with results at report@fotonower.com args[1337329739] : ('0.062116323844456485',) no score found for photo 1337329739 apple ('0.062116323844456485',) We are sending mail with results at report@fotonower.com args[1337329695] : ('0.062116323844456485',) no score found for photo 1337329695 apple ('0.062116323844456485',) We are sending mail with results at report@fotonower.com args[1337329664] : ('0.062116323844456485',) no score found for photo 1337329664 apple ('0.062116323844456485',) We are sending mail with results at report@fotonower.com refus_total : 0.062116323844456485 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20528860 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1337329752,1337330895,1337327016,1337328401,1337328993,1337329003,1337329664,1337328105,1337329744,1337329755,1337327015,1337462721,1337462725,1337462726,1337462736,1337462743,1337462747,1337462758,1337462769,1337462803) Found this number of photos: 20 begin to download photo : 1337329752 begin to download photo : 1337329003 begin to download photo : 1337327015 begin to download photo : 1337462743 download finish for photo 1337462743 begin to download photo : 1337462747 download finish for photo 1337329003 begin to download photo : 1337329664 download finish for photo 1337327015 begin to download photo : 1337462721 download finish for photo 1337329752 begin to download photo : 1337330895 download finish for photo 1337329664 begin to download photo : 1337328105 download finish for photo 1337462747 begin to download photo : 1337462758 download finish for photo 1337330895 begin to download photo : 1337327016 download finish for photo 1337462721 begin to download photo : 1337462725 download finish for photo 1337328105 begin to download photo : 1337329744 download finish for photo 1337462758 begin to download photo : 1337462769 download finish for photo 1337329744 begin to download photo : 1337329755 download finish for photo 1337327016 begin to download photo : 1337328401 download finish for photo 1337462725 begin to download photo : 1337462726 download finish for photo 1337462769 begin to download photo : 1337462803 download finish for photo 1337462726 begin to download photo : 1337462736 download finish for photo 1337329755 download finish for photo 1337462803 download finish for photo 1337328401 begin to download photo : 1337328993 download finish for photo 1337462736 download finish for photo 1337328993 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20528860_14-02-2025_00_40_44.pdf results_Auto_P20528860_14-02-2025_00_40_44.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20528860_14-02-2025_00_40_44.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20528860','results_Auto_P20528860_14-02-2025_00_40_44.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20528860_14-02-2025_00_40_44.pdf','pdf','','1.18','0.062116323844456485') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1337462721, 1337330895, 1337330865, 1337329755, 1337329752, 1337329748, 1337329744, 1337329739, 1337329695, 1337329664] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337462721', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330895', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330865', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329755', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329752', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329748', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329744', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329739', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329695', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329664', None, None, None, None, None, '2584387') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.021313905715942383 save_final save missing photos in datou_result : time spend for datou_step_exec : 9.432143449783325 time spend to save output : 0.021485567092895508 total time spend for step 10 : 9.45362901687622 step11:split_time_score Fri Feb 14 00:40:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 22), ('11', 22)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 13022025 20528860 Nombre de photos uploadées : 44 / 23040 (0%) 13022025 20528860 Nombre de photos taguées (types de déchets): 0 / 44 (0%) 13022025 20528860 Nombre de photos taguées (volume) : 0 / 44 (0%) elapsed_time : load_data_split_time_score 3.5762786865234375e-06 elapsed_time : order_list_meta_photo_and_scores 8.58306884765625e-06 ???????????????????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0033195018768310547 elapsed_time : insert_dashboard_record_day_entry 0.03770756721496582 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528841 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528843 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528845 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528847 order by id desc limit 1 Qualite : 0.14136086604967552 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20503797_13-02-2025_09_15_29.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20503797 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20503797 AND mptpi.`type`=3594 To do find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20503798_13-02-2025_15_42_23.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20503798 order by id desc limit 1 Qualite : 0.14449458458151784 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20503799_13-02-2025_09_01_27.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20503799 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20503799 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528849 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528851 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528853 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528854 order by id desc limit 1 Qualite : 0.03872013921048173 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20528856_14-02-2025_00_31_47.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528856 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11524 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11533 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11532 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11528 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11528 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11531 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11531 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11578 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11578 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11527 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11526 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11526 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11535 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11527 doesn't seem to be define in the database( WARNING : type of input 3 of step 11526 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11524 doesn't seem to be define in the database( WARNING : type of input 2 of step 11528 doesn't seem to be define in the database( WARNING : output 1 of step 11524 have datatype=2 whereas input 1 of step 11531 have datatype=7 WARNING : type of output 2 of step 11531 doesn't seem to be define in the database( WARNING : type of input 1 of step 11525 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11527 have datatype=10 whereas input 3 of step 11534 have datatype=6 WARNING : type of input 2 of step 11578 doesn't seem to be define in the database( WARNING : output 1 of step 11525 have datatype=7 whereas input 2 of step 11578 have datatype=None WARNING : type of output 3 of step 11578 doesn't seem to be define in the database( WARNING : type of input 1 of step 11527 doesn't seem to be define in the database( WARNING : output 0 of step 11527 have datatype=10 whereas input 0 of step 11584 have datatype=18 WARNING : type of input 5 of step 11534 doesn't seem to be define in the database( WARNING : output 0 of step 11584 have datatype=11 whereas input 5 of step 11534 have datatype=None WARNING : type of output 1 of step 11533 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : type of output 1 of step 11532 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : output 0 of step 11531 have datatype=1 whereas input 0 of step 11525 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20528856 AND mptpi.`type`=4211 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528858 order by id desc limit 1 Qualite : 0.07891361257373916 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20528860_14-02-2025_00_40_44.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528860 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20528860 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528862 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528864 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528865 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528867 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528869 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528871 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528872 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528874 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528876 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528878 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528881 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528906 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528907 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528909 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528911 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528937 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528939 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528940 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528941 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528942 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528943 order by id desc limit 1 Qualite : 0.037230814385410545 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528944 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20528944 AND mptpi.`type`=3327 To do Qualite : 0.06743200877946215 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20528945_14-02-2025_00_17_47.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528945 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20528945 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528946 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528947 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528948 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528949 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528950 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528951 order by id desc limit 1 Qualite : 0.14551669770917627 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20528952_14-02-2025_00_32_16.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528952 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20528952 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528953 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528954 order by id desc limit 1 Qualite : 0.14252729540879808 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20528957_14-02-2025_00_34_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528957 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20528957 AND mptpi.`type`=3327 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528958 order by id desc limit 1 Qualite : 0.04225902608664615 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20528959_14-02-2025_00_29_29.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20528959 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20528959 AND mptpi.`type`=4207 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'13022025': {'nb_upload': 44, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1337462721, 1337330895, 1337330865, 1337329755, 1337329752, 1337329748, 1337329744, 1337329739, 1337329695, 1337329664] Looping around the photos to save general results len do output : 1 /20528860Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337462721', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330895', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337330865', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329755', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329752', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329748', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329744', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329739', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329695', None, None, None, None, None, '2584387') ('3459', None, None, None, None, None, None, None, '2584387') ('3459', None, '1337329664', None, None, None, None, None, '2584387') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.24138236045837402 save_final save missing photos in datou_result : time spend for datou_step_exec : 32.980647563934326 time spend to save output : 0.241713285446167 total time spend for step 11 : 33.22236084938049 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 135.68user 151.27system 7:04.20elapsed 67%CPU (0avgtext+0avgdata 4249884maxresident)k 500568inputs+146440outputs (17126major+11395117minor)pagefaults 0swaps