python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 116951 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 17>10 please execute split_portfolio.py -i 21131115 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2646484'] with mtr_portfolio_ids : ['21131115'] and first list_photo_ids : [] new path : /proc/116951/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.9019734859466553 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Mar 7 00:12:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4165 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-07 00:12:31.551199: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-07 00:12:31.579246: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-07 00:12:31.581906: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f3490000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-07 00:12:31.581970: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-07 00:12:31.586306: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-07 00:12:31.769988: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xc6b2930 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-07 00:12:31.770047: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-07 00:12:31.771241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 00:12:31.771660: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 00:12:31.774728: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 00:12:31.777841: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 00:12:31.778317: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 00:12:31.781953: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 00:12:31.783735: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 00:12:31.789898: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 00:12:31.791054: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 00:12:31.791131: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 00:12:31.791711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-07 00:12:31.791726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-07 00:12:31.791735: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-07 00:12:31.792739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3713 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-07 00:12:32.109400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 00:12:32.109545: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 00:12:32.109565: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 00:12:32.109581: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 00:12:32.109597: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 00:12:32.109612: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 00:12:32.109626: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 00:12:32.109642: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 00:12:32.110534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 00:12:32.111748: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 00:12:32.111784: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 00:12:32.111799: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 00:12:32.111813: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 00:12:32.111827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 00:12:32.111841: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 00:12:32.111855: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 00:12:32.111869: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 00:12:32.112770: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 00:12:32.112811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-07 00:12:32.112821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-07 00:12:32.112829: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-07 00:12:32.113730: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3713 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-07 00:12:47.090990: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 00:12:47.330547: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 00:12:49.295692: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.13G (3358457856 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:12:50.128608: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:12:50.128724: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.02358 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 17 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.92312 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 24 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 28 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 17 Detection mask done ! Trying to reset tf kernel 117361 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 309 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4386 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.6913073062896729 nb_pixel_total : 4883292 time to create 1 rle with new method : 1.1580710411071777 length of segment : 4776 time for calcul the mask position with numpy : 0.0004975795745849609 nb_pixel_total : 38431 time to create 1 rle with old method : 0.042485713958740234 length of segment : 233 time for calcul the mask position with numpy : 0.025408506393432617 nb_pixel_total : 831885 time to create 1 rle with new method : 0.5575668811798096 length of segment : 2975 time for calcul the mask position with numpy : 0.0031595230102539062 nb_pixel_total : 212393 time to create 1 rle with new method : 0.008437633514404297 length of segment : 1090 time for calcul the mask position with numpy : 0.0025315284729003906 nb_pixel_total : 195091 time to create 1 rle with new method : 0.005781650543212891 length of segment : 606 time for calcul the mask position with numpy : 0.002101898193359375 nb_pixel_total : 72724 time to create 1 rle with old method : 0.0815134048461914 length of segment : 401 time for calcul the mask position with numpy : 0.0003459453582763672 nb_pixel_total : 9680 time to create 1 rle with old method : 0.011622905731201172 length of segment : 131 time for calcul the mask position with numpy : 0.001049041748046875 nb_pixel_total : 31454 time to create 1 rle with old method : 0.03727555274963379 length of segment : 281 time for calcul the mask position with numpy : 0.0006089210510253906 nb_pixel_total : 18597 time to create 1 rle with old method : 0.021421194076538086 length of segment : 160 time for calcul the mask position with numpy : 0.007558107376098633 nb_pixel_total : 212039 time to create 1 rle with new method : 0.011336326599121094 length of segment : 900 time for calcul the mask position with numpy : 0.002981424331665039 nb_pixel_total : 121350 time to create 1 rle with old method : 0.1350719928741455 length of segment : 416 time for calcul the mask position with numpy : 0.00043272972106933594 nb_pixel_total : 10552 time to create 1 rle with old method : 0.012323856353759766 length of segment : 177 time for calcul the mask position with numpy : 8.106231689453125e-05 nb_pixel_total : 2769 time to create 1 rle with old method : 0.0034427642822265625 length of segment : 46 time for calcul the mask position with numpy : 0.0014755725860595703 nb_pixel_total : 66003 time to create 1 rle with old method : 0.07662272453308105 length of segment : 285 time for calcul the mask position with numpy : 0.0006434917449951172 nb_pixel_total : 13809 time to create 1 rle with old method : 0.016006946563720703 length of segment : 224 time for calcul the mask position with numpy : 0.0006208419799804688 nb_pixel_total : 22085 time to create 1 rle with old method : 0.025876522064208984 length of segment : 144 time for calcul the mask position with numpy : 0.0026428699493408203 nb_pixel_total : 160647 time to create 1 rle with new method : 0.007281303405761719 length of segment : 488 time for calcul the mask position with numpy : 0.0006287097930908203 nb_pixel_total : 36143 time to create 1 rle with old method : 0.04146242141723633 length of segment : 260 time for calcul the mask position with numpy : 0.008939027786254883 nb_pixel_total : 332268 time to create 1 rle with new method : 0.020527362823486328 length of segment : 1217 time for calcul the mask position with numpy : 0.0007295608520507812 nb_pixel_total : 26920 time to create 1 rle with old method : 0.03074049949645996 length of segment : 251 time for calcul the mask position with numpy : 0.3647141456604004 nb_pixel_total : 5169399 time to create 1 rle with new method : 1.7470407485961914 length of segment : 3206 time for calcul the mask position with numpy : 0.0004954338073730469 nb_pixel_total : 32166 time to create 1 rle with old method : 0.03627967834472656 length of segment : 275 time for calcul the mask position with numpy : 0.00023174285888671875 nb_pixel_total : 10173 time to create 1 rle with old method : 0.012062311172485352 length of segment : 50 time for calcul the mask position with numpy : 0.0012674331665039062 nb_pixel_total : 95649 time to create 1 rle with old method : 0.10918951034545898 length of segment : 352 time for calcul the mask position with numpy : 0.0014781951904296875 nb_pixel_total : 56976 time to create 1 rle with old method : 0.06410527229309082 length of segment : 426 time for calcul the mask position with numpy : 0.3996710777282715 nb_pixel_total : 3126062 time to create 1 rle with new method : 0.6435031890869141 length of segment : 2715 time for calcul the mask position with numpy : 0.0014467239379882812 nb_pixel_total : 102189 time to create 1 rle with old method : 0.12098526954650879 length of segment : 355 time for calcul the mask position with numpy : 0.014811038970947266 nb_pixel_total : 690548 time to create 1 rle with new method : 0.16395354270935059 length of segment : 1212 time for calcul the mask position with numpy : 0.00031304359436035156 nb_pixel_total : 4075 time to create 1 rle with old method : 0.008192777633666992 length of segment : 72 time for calcul the mask position with numpy : 0.0001773834228515625 nb_pixel_total : 6707 time to create 1 rle with old method : 0.008583784103393555 length of segment : 165 time for calcul the mask position with numpy : 0.0024156570434570312 nb_pixel_total : 130540 time to create 1 rle with old method : 0.1467750072479248 length of segment : 655 time for calcul the mask position with numpy : 0.0036630630493164062 nb_pixel_total : 227825 time to create 1 rle with new method : 0.009270668029785156 length of segment : 781 time for calcul the mask position with numpy : 0.0009195804595947266 nb_pixel_total : 27961 time to create 1 rle with old method : 0.031534671783447266 length of segment : 229 time for calcul the mask position with numpy : 0.0002598762512207031 nb_pixel_total : 5037 time to create 1 rle with old method : 0.006336212158203125 length of segment : 99 time for calcul the mask position with numpy : 0.0011513233184814453 nb_pixel_total : 28163 time to create 1 rle with old method : 0.03445577621459961 length of segment : 257 time for calcul the mask position with numpy : 0.0013575553894042969 nb_pixel_total : 51769 time to create 1 rle with old method : 0.07756757736206055 length of segment : 259 time for calcul the mask position with numpy : 0.00019669532775878906 nb_pixel_total : 3919 time to create 1 rle with old method : 0.0047223567962646484 length of segment : 50 time for calcul the mask position with numpy : 0.00021505355834960938 nb_pixel_total : 3698 time to create 1 rle with old method : 0.0043544769287109375 length of segment : 77 time for calcul the mask position with numpy : 0.00814366340637207 nb_pixel_total : 351503 time to create 1 rle with new method : 0.011137008666992188 length of segment : 676 time for calcul the mask position with numpy : 0.004068613052368164 nb_pixel_total : 122171 time to create 1 rle with old method : 0.14119625091552734 length of segment : 504 time for calcul the mask position with numpy : 0.00370025634765625 nb_pixel_total : 149189 time to create 1 rle with old method : 0.1690845489501953 length of segment : 396 time for calcul the mask position with numpy : 0.00692296028137207 nb_pixel_total : 269975 time to create 1 rle with new method : 0.00945734977722168 length of segment : 590 time for calcul the mask position with numpy : 0.003080129623413086 nb_pixel_total : 64002 time to create 1 rle with old method : 0.07290315628051758 length of segment : 473 time for calcul the mask position with numpy : 0.0028045177459716797 nb_pixel_total : 100226 time to create 1 rle with old method : 0.13133740425109863 length of segment : 670 time for calcul the mask position with numpy : 0.0012745857238769531 nb_pixel_total : 38877 time to create 1 rle with old method : 0.04800248146057129 length of segment : 238 time for calcul the mask position with numpy : 9.393692016601562e-05 nb_pixel_total : 1586 time to create 1 rle with old method : 0.0019714832305908203 length of segment : 37 time for calcul the mask position with numpy : 0.0029501914978027344 nb_pixel_total : 73548 time to create 1 rle with old method : 0.08736300468444824 length of segment : 432 time for calcul the mask position with numpy : 0.0024061203002929688 nb_pixel_total : 95493 time to create 1 rle with old method : 0.11280989646911621 length of segment : 250 time for calcul the mask position with numpy : 0.0002868175506591797 nb_pixel_total : 4255 time to create 1 rle with old method : 0.0055141448974609375 length of segment : 50 time for calcul the mask position with numpy : 0.0001499652862548828 nb_pixel_total : 1022 time to create 1 rle with old method : 0.0013206005096435547 length of segment : 53 time for calcul the mask position with numpy : 0.002356290817260742 nb_pixel_total : 53891 time to create 1 rle with old method : 0.06045174598693848 length of segment : 422 time for calcul the mask position with numpy : 0.0020780563354492188 nb_pixel_total : 50572 time to create 1 rle with old method : 0.05823254585266113 length of segment : 287 time for calcul the mask position with numpy : 0.0003046989440917969 nb_pixel_total : 6887 time to create 1 rle with old method : 0.008147478103637695 length of segment : 135 time for calcul the mask position with numpy : 0.004422903060913086 nb_pixel_total : 135862 time to create 1 rle with old method : 0.15215039253234863 length of segment : 534 time for calcul the mask position with numpy : 0.00029921531677246094 nb_pixel_total : 2708 time to create 1 rle with old method : 0.0032825469970703125 length of segment : 117 time for calcul the mask position with numpy : 0.0012011528015136719 nb_pixel_total : 18478 time to create 1 rle with old method : 0.022243738174438477 length of segment : 336 time for calcul the mask position with numpy : 0.0004942417144775391 nb_pixel_total : 5761 time to create 1 rle with old method : 0.006840944290161133 length of segment : 160 time for calcul the mask position with numpy : 0.0005795955657958984 nb_pixel_total : 17002 time to create 1 rle with old method : 0.020907163619995117 length of segment : 199 time for calcul the mask position with numpy : 0.0004811286926269531 nb_pixel_total : 6092 time to create 1 rle with old method : 0.010133743286132812 length of segment : 179 time for calcul the mask position with numpy : 0.0062215328216552734 nb_pixel_total : 126839 time to create 1 rle with old method : 0.15140199661254883 length of segment : 632 time for calcul the mask position with numpy : 0.0018689632415771484 nb_pixel_total : 56221 time to create 1 rle with old method : 0.06396961212158203 length of segment : 269 time for calcul the mask position with numpy : 0.0014529228210449219 nb_pixel_total : 49572 time to create 1 rle with old method : 0.05579662322998047 length of segment : 325 time for calcul the mask position with numpy : 0.001590728759765625 nb_pixel_total : 67775 time to create 1 rle with old method : 0.07740116119384766 length of segment : 314 time for calcul the mask position with numpy : 0.0010950565338134766 nb_pixel_total : 51491 time to create 1 rle with old method : 0.07071042060852051 length of segment : 264 time for calcul the mask position with numpy : 0.002389192581176758 nb_pixel_total : 115165 time to create 1 rle with old method : 0.12763547897338867 length of segment : 418 time for calcul the mask position with numpy : 0.000858306884765625 nb_pixel_total : 20212 time to create 1 rle with old method : 0.023866653442382812 length of segment : 239 time for calcul the mask position with numpy : 0.00036787986755371094 nb_pixel_total : 18773 time to create 1 rle with old method : 0.02303147315979004 length of segment : 218 time for calcul the mask position with numpy : 0.0001838207244873047 nb_pixel_total : 2503 time to create 1 rle with old method : 0.0031180381774902344 length of segment : 66 time for calcul the mask position with numpy : 0.0005254745483398438 nb_pixel_total : 8186 time to create 1 rle with old method : 0.01005697250366211 length of segment : 109 time for calcul the mask position with numpy : 0.0010483264923095703 nb_pixel_total : 28945 time to create 1 rle with old method : 0.03320908546447754 length of segment : 229 time for calcul the mask position with numpy : 0.0016145706176757812 nb_pixel_total : 114566 time to create 1 rle with old method : 0.15313720703125 length of segment : 411 time for calcul the mask position with numpy : 0.000171661376953125 nb_pixel_total : 2786 time to create 1 rle with old method : 0.0035228729248046875 length of segment : 48 time for calcul the mask position with numpy : 0.001331329345703125 nb_pixel_total : 40140 time to create 1 rle with old method : 0.046090126037597656 length of segment : 499 time for calcul the mask position with numpy : 0.0036847591400146484 nb_pixel_total : 162107 time to create 1 rle with new method : 0.0052547454833984375 length of segment : 563 time for calcul the mask position with numpy : 0.00036406517028808594 nb_pixel_total : 9290 time to create 1 rle with old method : 0.011543989181518555 length of segment : 122 time for calcul the mask position with numpy : 0.00016498565673828125 nb_pixel_total : 1681 time to create 1 rle with old method : 0.0021448135375976562 length of segment : 53 time for calcul the mask position with numpy : 0.00168609619140625 nb_pixel_total : 31738 time to create 1 rle with old method : 0.0364229679107666 length of segment : 210 time for calcul the mask position with numpy : 0.0004603862762451172 nb_pixel_total : 8300 time to create 1 rle with old method : 0.013321399688720703 length of segment : 128 time for calcul the mask position with numpy : 0.00017976760864257812 nb_pixel_total : 1363 time to create 1 rle with old method : 0.0019202232360839844 length of segment : 67 time for calcul the mask position with numpy : 0.005543708801269531 nb_pixel_total : 188852 time to create 1 rle with new method : 0.010096311569213867 length of segment : 524 time for calcul the mask position with numpy : 0.0010480880737304688 nb_pixel_total : 24876 time to create 1 rle with old method : 0.028963327407836914 length of segment : 264 time for calcul the mask position with numpy : 0.0007991790771484375 nb_pixel_total : 15848 time to create 1 rle with old method : 0.0184478759765625 length of segment : 221 time for calcul the mask position with numpy : 0.0013432502746582031 nb_pixel_total : 36412 time to create 1 rle with old method : 0.04165220260620117 length of segment : 263 time for calcul the mask position with numpy : 0.0006871223449707031 nb_pixel_total : 14067 time to create 1 rle with old method : 0.016834497451782227 length of segment : 170 time for calcul the mask position with numpy : 0.00030517578125 nb_pixel_total : 3862 time to create 1 rle with old method : 0.005666255950927734 length of segment : 82 time for calcul the mask position with numpy : 0.0009396076202392578 nb_pixel_total : 19393 time to create 1 rle with old method : 0.03168058395385742 length of segment : 161 time for calcul the mask position with numpy : 0.0016794204711914062 nb_pixel_total : 29870 time to create 1 rle with old method : 0.03517341613769531 length of segment : 368 time for calcul the mask position with numpy : 0.00037217140197753906 nb_pixel_total : 8256 time to create 1 rle with old method : 0.00986170768737793 length of segment : 95 time for calcul the mask position with numpy : 0.0010197162628173828 nb_pixel_total : 36899 time to create 1 rle with old method : 0.04195833206176758 length of segment : 311 time for calcul the mask position with numpy : 0.007210969924926758 nb_pixel_total : 185131 time to create 1 rle with new method : 0.009105920791625977 length of segment : 513 time for calcul the mask position with numpy : 0.00022721290588378906 nb_pixel_total : 5868 time to create 1 rle with old method : 0.007027387619018555 length of segment : 85 time for calcul the mask position with numpy : 0.00014662742614746094 nb_pixel_total : 2197 time to create 1 rle with old method : 0.0027136802673339844 length of segment : 48 time for calcul the mask position with numpy : 0.0008971691131591797 nb_pixel_total : 31190 time to create 1 rle with old method : 0.034688711166381836 length of segment : 225 time for calcul the mask position with numpy : 0.00029587745666503906 nb_pixel_total : 8224 time to create 1 rle with old method : 0.009841203689575195 length of segment : 115 time for calcul the mask position with numpy : 0.0013189315795898438 nb_pixel_total : 46144 time to create 1 rle with old method : 0.05536389350891113 length of segment : 282 time for calcul the mask position with numpy : 0.00021648406982421875 nb_pixel_total : 2273 time to create 1 rle with old method : 0.002904176712036133 length of segment : 68 time for calcul the mask position with numpy : 0.0003535747528076172 nb_pixel_total : 7095 time to create 1 rle with old method : 0.008428812026977539 length of segment : 100 time for calcul the mask position with numpy : 0.0004341602325439453 nb_pixel_total : 8569 time to create 1 rle with old method : 0.00991678237915039 length of segment : 103 time for calcul the mask position with numpy : 0.0005259513854980469 nb_pixel_total : 38258 time to create 1 rle with old method : 0.04384112358093262 length of segment : 239 time for calcul the mask position with numpy : 0.00040841102600097656 nb_pixel_total : 19877 time to create 1 rle with old method : 0.02270984649658203 length of segment : 199 time for calcul the mask position with numpy : 0.0007257461547851562 nb_pixel_total : 12786 time to create 1 rle with old method : 0.015430450439453125 length of segment : 206 time for calcul the mask position with numpy : 0.0013241767883300781 nb_pixel_total : 23896 time to create 1 rle with old method : 0.028326988220214844 length of segment : 209 time for calcul the mask position with numpy : 0.0010733604431152344 nb_pixel_total : 19140 time to create 1 rle with old method : 0.023105382919311523 length of segment : 200 time for calcul the mask position with numpy : 0.0007867813110351562 nb_pixel_total : 23618 time to create 1 rle with old method : 0.02795553207397461 length of segment : 197 time for calcul the mask position with numpy : 0.0009162425994873047 nb_pixel_total : 30412 time to create 1 rle with old method : 0.034896135330200195 length of segment : 275 time for calcul the mask position with numpy : 0.0007445812225341797 nb_pixel_total : 10185 time to create 1 rle with old method : 0.011676788330078125 length of segment : 285 time for calcul the mask position with numpy : 0.035687923431396484 nb_pixel_total : 996823 time to create 1 rle with new method : 0.15728282928466797 length of segment : 2865 time for calcul the mask position with numpy : 0.00045561790466308594 nb_pixel_total : 12178 time to create 1 rle with old method : 0.014202117919921875 length of segment : 100 time for calcul the mask position with numpy : 0.0008103847503662109 nb_pixel_total : 52361 time to create 1 rle with old method : 0.06406450271606445 length of segment : 238 time for calcul the mask position with numpy : 0.0026543140411376953 nb_pixel_total : 100873 time to create 1 rle with old method : 0.12019538879394531 length of segment : 525 time for calcul the mask position with numpy : 0.0002200603485107422 nb_pixel_total : 2813 time to create 1 rle with old method : 0.004021644592285156 length of segment : 44 time for calcul the mask position with numpy : 0.006479024887084961 nb_pixel_total : 241225 time to create 1 rle with new method : 0.009557962417602539 length of segment : 585 time for calcul the mask position with numpy : 0.006169319152832031 nb_pixel_total : 289446 time to create 1 rle with new method : 0.01649188995361328 length of segment : 685 time for calcul the mask position with numpy : 0.0032842159271240234 nb_pixel_total : 105713 time to create 1 rle with old method : 0.1197502613067627 length of segment : 580 time for calcul the mask position with numpy : 0.0018305778503417969 nb_pixel_total : 81807 time to create 1 rle with old method : 0.0903923511505127 length of segment : 406 time for calcul the mask position with numpy : 0.002798318862915039 nb_pixel_total : 129209 time to create 1 rle with old method : 0.17040157318115234 length of segment : 597 time for calcul the mask position with numpy : 0.0016922950744628906 nb_pixel_total : 72106 time to create 1 rle with old method : 0.10648822784423828 length of segment : 429 time for calcul the mask position with numpy : 0.0005223751068115234 nb_pixel_total : 7676 time to create 1 rle with old method : 0.011140823364257812 length of segment : 211 time for calcul the mask position with numpy : 0.0005018711090087891 nb_pixel_total : 27956 time to create 1 rle with old method : 0.03408980369567871 length of segment : 209 time for calcul the mask position with numpy : 0.0023698806762695312 nb_pixel_total : 107992 time to create 1 rle with old method : 0.12299299240112305 length of segment : 572 time for calcul the mask position with numpy : 0.0028183460235595703 nb_pixel_total : 122971 time to create 1 rle with old method : 0.13772988319396973 length of segment : 432 time for calcul the mask position with numpy : 0.0006322860717773438 nb_pixel_total : 11214 time to create 1 rle with old method : 0.013218879699707031 length of segment : 267 time for calcul the mask position with numpy : 0.02806854248046875 nb_pixel_total : 704420 time to create 1 rle with new method : 0.17301082611083984 length of segment : 1985 time for calcul the mask position with numpy : 0.0011072158813476562 nb_pixel_total : 67017 time to create 1 rle with old method : 0.07522797584533691 length of segment : 400 time for calcul the mask position with numpy : 0.0011875629425048828 nb_pixel_total : 68115 time to create 1 rle with old method : 0.07692766189575195 length of segment : 317 time for calcul the mask position with numpy : 0.00015664100646972656 nb_pixel_total : 1255 time to create 1 rle with old method : 0.0015227794647216797 length of segment : 62 time for calcul the mask position with numpy : 0.005156993865966797 nb_pixel_total : 247943 time to create 1 rle with new method : 0.00872802734375 length of segment : 591 time for calcul the mask position with numpy : 0.0004260540008544922 nb_pixel_total : 11598 time to create 1 rle with old method : 0.013460159301757812 length of segment : 103 time for calcul the mask position with numpy : 0.00010061264038085938 nb_pixel_total : 3900 time to create 1 rle with old method : 0.004901885986328125 length of segment : 59 time for calcul the mask position with numpy : 0.0024213790893554688 nb_pixel_total : 98100 time to create 1 rle with old method : 0.11141395568847656 length of segment : 594 time for calcul the mask position with numpy : 0.0009915828704833984 nb_pixel_total : 50246 time to create 1 rle with old method : 0.05781865119934082 length of segment : 487 time for calcul the mask position with numpy : 0.002849578857421875 nb_pixel_total : 82187 time to create 1 rle with old method : 0.09373974800109863 length of segment : 436 time for calcul the mask position with numpy : 0.008556842803955078 nb_pixel_total : 445872 time to create 1 rle with new method : 0.030974149703979492 length of segment : 2086 time for calcul the mask position with numpy : 0.001817941665649414 nb_pixel_total : 96635 time to create 1 rle with old method : 0.10826992988586426 length of segment : 381 time for calcul the mask position with numpy : 7.963180541992188e-05 nb_pixel_total : 1157 time to create 1 rle with old method : 0.0014238357543945312 length of segment : 57 time spent for convertir_results : 24.786478281021118 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 135 chid ids of type : 3663 Number RLEs to save : 60329 save missing photos in datou_result : time spend for datou_step_exec : 91.433664560318 time spend to save output : 3.6378819942474365 total time spend for step 1 : 95.07154655456543 step2:crop_condition Fri Mar 7 00:14:03 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 135 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 36 About to insert : list_path_to_insert length 36 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 13 About to insert : list_path_to_insert length 13 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987, 1341265981, 1341265975, 1341265971] Looping around the photos to save general results len do output : 78 /-3696598122Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598147Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598169Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598180Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598181Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598190Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598175Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598184Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598172Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598093Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598113Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598111Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598135Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598145Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598127Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598196Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598206Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598211Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598225Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598109Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598106Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598134Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598139Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598156Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598171Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598096Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598095Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598110Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598097Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598102Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598101Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598105Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598115Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598123Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598142Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598138Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598136Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598132Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598153Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598155Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598161Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598152Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598162Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598151Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598154Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598165Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598164Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598174Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598186Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598189Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598193Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598177Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598200Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598201Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598212Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598205Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598207Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598210Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598216Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598223Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598221Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598100Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598103Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598098Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598104Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598114Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598118Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598129Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598158Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598168Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598166Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598191Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598179Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598188Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598131Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598208Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598215Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598157Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266663', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266213', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266207', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266194', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266186', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265999', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265987', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265981', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265975', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265971', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 244 time used for this insertion : 0.03363966941833496 save_final save missing photos in datou_result : time spend for datou_step_exec : 43.232994079589844 time spend to save output : 0.035797834396362305 total time spend for step 2 : 43.268791913986206 step3:thcl Fri Mar 7 00:14:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.007734537124633789 time to convert the images to numpy array : 0.07299637794494629 time to import caffe and check if the image exist : 0.011310338973999023 time to convert the images to numpy array : 0.7513160705566406 time to import caffe and check if the image exist : 0.006793022155761719 time to convert the images to numpy array : 0.7668724060058594 time to import caffe and check if the image exist : 0.009050369262695312 time to convert the images to numpy array : 0.7825524806976318 time to import caffe and check if the image exist : 0.010278940200805664 time to convert the images to numpy array : 0.7811741828918457 time to import caffe and check if the image exist : 0.011382341384887695 time to convert the images to numpy array : 0.7861273288726807 time to import caffe and check if the image exist : 0.01163029670715332 time to convert the images to numpy array : 0.7944743633270264 time to import caffe and check if the image exist : 0.013822793960571289 time to convert the images to numpy array : 0.7951982021331787 time to import caffe and check if the image exist : 0.010487079620361328 time to convert the images to numpy array : 0.8073229789733887 time to import caffe and check if the image exist : 0.014333009719848633 time to convert the images to numpy array : 1.1254963874816895 total time to convert the images to numpy array : 1.4469504356384277 list photo_ids error: [] list photo_ids correct : [-3696598104, -3696598114, -3696598118, -3696598129, -3696598158, -3696598168, -3696598166, -3696598191, -3696598179, -3696598188, -3696598131, -3696598208, -3696598215, -3696598157, -3696598122, -3696598147, -3696598169, -3696598180, -3696598181, -3696598190, -3696598175, -3696598184, -3696598172, -3696598093, -3696598113, -3696598111, -3696598135, -3696598145, -3696598127, -3696598196, -3696598207, -3696598210, -3696598216, -3696598223, -3696598221, -3696598100, -3696598103, -3696598098, -3696598186, -3696598189, -3696598193, -3696598177, -3696598200, -3696598201, -3696598212, -3696598205, -3696598206, -3696598211, -3696598225, -3696598109, -3696598106, -3696598134, -3696598139, -3696598156, -3696598161, -3696598152, -3696598162, -3696598151, -3696598154, -3696598165, -3696598164, -3696598174, -3696598115, -3696598123, -3696598142, -3696598138, -3696598136, -3696598132, -3696598153, -3696598155, -3696598171, -3696598096, -3696598095, -3696598110, -3696598097, -3696598102, -3696598101, -3696598105] number of photos to traite : 78 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 111 wait 20 seconds l 3637 free memory gpu now : 111 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3195 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.9469704627990723 time used to do the prediction : 0.20882391929626465 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3193 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3193 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.9706826210021973 time used to do the prediction : 0.15525031089782715 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.5510787963867188e-05 save missing photos in datou_result : time spend for datou_step_exec : 30.64974570274353 time spend to save output : 0.0006427764892578125 total time spend for step 3 : 30.650388479232788 step4:merge_mask_thcl_custom Fri Mar 7 00:15:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 135 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987, 1341265981, 1341265975, 1341265971] Looping around the photos to save general results len do output : 10 /1341266663Didn't retrieve data .Didn't retrieve data . /1341266213Didn't retrieve data .Didn't retrieve data . /1341266207Didn't retrieve data .Didn't retrieve data . /1341266194Didn't retrieve data .Didn't retrieve data . /1341266186Didn't retrieve data .Didn't retrieve data . /1341265999Didn't retrieve data .Didn't retrieve data . /1341265987Didn't retrieve data .Didn't retrieve data . /1341265981Didn't retrieve data .Didn't retrieve data . /1341265975Didn't retrieve data .Didn't retrieve data . /1341265971Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266663', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266213', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266207', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266194', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266186', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265999', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265987', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265981', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265975', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265971', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014168024063110352 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.0570981502532959 time spend to save output : 0.014816761016845703 total time spend for step 4 : 0.0719149112701416 step5:rle_unique_nms_with_priority Fri Mar 7 00:15:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 4 nb_hashtags : 3 time to prepare the origin masks : 7.9224114418029785 time for calcul the mask position with numpy : 0.06838488578796387 nb_pixel_total : 1410164 time to create 1 rle with new method : 0.9954566955566406 time for calcul the mask position with numpy : 0.03646135330200195 nb_pixel_total : 46711 time to create 1 rle with old method : 0.05286431312561035 time for calcul the mask position with numpy : 0.03510689735412598 nb_pixel_total : 68634 time to create 1 rle with old method : 0.07933902740478516 time for calcul the mask position with numpy : 0.021235227584838867 nb_pixel_total : 32963 time to create 1 rle with old method : 0.035552024841308594 time for calcul the mask position with numpy : 1.3692142963409424 nb_pixel_total : 5491768 time to create 1 rle with new method : 1.1291999816894531 create new chi : 3.882965326309204 time to delete rle : 0.014111995697021484 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 11943 TO DO : save crop sub photo not yet done ! save time : 0.7355360984802246 nb_obj : 13 nb_hashtags : 5 time to prepare the origin masks : 14.54323673248291 time for calcul the mask position with numpy : 0.9303672313690186 nb_pixel_total : 6113785 time to create 1 rle with new method : 1.1292428970336914 time for calcul the mask position with numpy : 0.03526139259338379 nb_pixel_total : 63315 time to create 1 rle with old method : 0.07186150550842285 time for calcul the mask position with numpy : 0.03317666053771973 nb_pixel_total : 65820 time to create 1 rle with old method : 0.07522273063659668 time for calcul the mask position with numpy : 0.026164770126342773 nb_pixel_total : 72951 time to create 1 rle with old method : 0.08631134033203125 time for calcul the mask position with numpy : 0.03220629692077637 nb_pixel_total : 18538 time to create 1 rle with old method : 0.021602630615234375 time for calcul the mask position with numpy : 0.0234987735748291 nb_pixel_total : 9641 time to create 1 rle with old method : 0.012243986129760742 time for calcul the mask position with numpy : 0.022544384002685547 nb_pixel_total : 2752 time to create 1 rle with old method : 0.003244638442993164 time for calcul the mask position with numpy : 0.022838830947875977 nb_pixel_total : 160340 time to create 1 rle with new method : 1.339212417602539 time for calcul the mask position with numpy : 0.02086639404296875 nb_pixel_total : 10511 time to create 1 rle with old method : 0.012134552001953125 time for calcul the mask position with numpy : 0.02172994613647461 nb_pixel_total : 13777 time to create 1 rle with old method : 0.015197277069091797 time for calcul the mask position with numpy : 0.02550363540649414 nb_pixel_total : 330206 time to create 1 rle with new method : 0.5274679660797119 time for calcul the mask position with numpy : 0.021701812744140625 nb_pixel_total : 31400 time to create 1 rle with old method : 0.0347437858581543 time for calcul the mask position with numpy : 0.021967172622680664 nb_pixel_total : 121202 time to create 1 rle with old method : 0.13395237922668457 time for calcul the mask position with numpy : 0.021825790405273438 nb_pixel_total : 36002 time to create 1 rle with old method : 0.039313316345214844 create new chi : 4.850039958953857 time to delete rle : 0.0013926029205322266 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 11850 TO DO : save crop sub photo not yet done ! save time : 0.6932597160339355 nb_obj : 5 nb_hashtags : 4 time to prepare the origin masks : 6.8809144496917725 time for calcul the mask position with numpy : 0.04744219779968262 nb_pixel_total : 1769710 time to create 1 rle with new method : 0.4924478530883789 time for calcul the mask position with numpy : 0.020923852920532227 nb_pixel_total : 16341 time to create 1 rle with old method : 0.018023014068603516 time for calcul the mask position with numpy : 0.02063608169555664 nb_pixel_total : 8522 time to create 1 rle with old method : 0.009672164916992188 time for calcul the mask position with numpy : 1.3652377128601074 nb_pixel_total : 5133433 time to create 1 rle with new method : 1.8639883995056152 time for calcul the mask position with numpy : 0.0260007381439209 nb_pixel_total : 26748 time to create 1 rle with old method : 0.037929534912109375 time for calcul the mask position with numpy : 0.02389383316040039 nb_pixel_total : 95486 time to create 1 rle with old method : 0.10480380058288574 create new chi : 4.095718860626221 time to delete rle : 0.0010688304901123047 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 9827 TO DO : save crop sub photo not yet done ! save time : 0.6386220455169678 nb_obj : 8 nb_hashtags : 4 time to prepare the origin masks : 8.512514591217041 time for calcul the mask position with numpy : 0.4014606475830078 nb_pixel_total : 3395033 time to create 1 rle with new method : 1.7083983421325684 time for calcul the mask position with numpy : 0.03491830825805664 nb_pixel_total : 102176 time to create 1 rle with old method : 0.1329514980316162 time for calcul the mask position with numpy : 0.03802895545959473 nb_pixel_total : 109793 time to create 1 rle with old method : 0.11942720413208008 time for calcul the mask position with numpy : 0.03831219673156738 nb_pixel_total : 101970 time to create 1 rle with old method : 0.11112737655639648 time for calcul the mask position with numpy : 0.035764455795288086 nb_pixel_total : 5480 time to create 1 rle with old method : 0.0064296722412109375 time for calcul the mask position with numpy : 0.03465104103088379 nb_pixel_total : 4062 time to create 1 rle with old method : 0.004529237747192383 time for calcul the mask position with numpy : 0.03603363037109375 nb_pixel_total : 152975 time to create 1 rle with new method : 1.3820648193359375 time for calcul the mask position with numpy : 0.5296432971954346 nb_pixel_total : 3121989 time to create 1 rle with new method : 1.7435922622680664 time for calcul the mask position with numpy : 0.03692126274108887 nb_pixel_total : 56762 time to create 1 rle with old method : 0.08152151107788086 create new chi : 6.5672197341918945 time to delete rle : 0.0016753673553466797 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 12829 TO DO : save crop sub photo not yet done ! save time : 0.7542619705200195 nb_obj : 21 nb_hashtags : 6 time to prepare the origin masks : 22.97408890724182 time for calcul the mask position with numpy : 0.7675316333770752 nb_pixel_total : 5514284 time to create 1 rle with new method : 1.4917845726013184 time for calcul the mask position with numpy : 0.039537906646728516 nb_pixel_total : 73373 time to create 1 rle with old method : 0.07808470726013184 time for calcul the mask position with numpy : 0.04432559013366699 nb_pixel_total : 121987 time to create 1 rle with old method : 0.13143396377563477 time for calcul the mask position with numpy : 0.04557609558105469 nb_pixel_total : 148661 time to create 1 rle with old method : 0.15642452239990234 time for calcul the mask position with numpy : 0.039438724517822266 nb_pixel_total : 6850 time to create 1 rle with old method : 0.19697880744934082 time for calcul the mask position with numpy : 0.03776264190673828 nb_pixel_total : 1012 time to create 1 rle with old method : 0.0011904239654541016 time for calcul the mask position with numpy : 0.022731542587280273 nb_pixel_total : 3683 time to create 1 rle with old method : 0.0038442611694335938 time for calcul the mask position with numpy : 0.022609710693359375 nb_pixel_total : 4220 time to create 1 rle with old method : 0.004734992980957031 time for calcul the mask position with numpy : 0.024071931838989258 nb_pixel_total : 1575 time to create 1 rle with old method : 0.0016715526580810547 time for calcul the mask position with numpy : 0.024693965911865234 nb_pixel_total : 100212 time to create 1 rle with old method : 0.11208319664001465 time for calcul the mask position with numpy : 0.023989200592041016 nb_pixel_total : 51668 time to create 1 rle with old method : 0.055133819580078125 time for calcul the mask position with numpy : 0.03576970100402832 nb_pixel_total : 3665 time to create 1 rle with old method : 0.004011869430541992 time for calcul the mask position with numpy : 0.039888858795166016 nb_pixel_total : 63790 time to create 1 rle with old method : 0.06916952133178711 time for calcul the mask position with numpy : 0.04294180870056152 nb_pixel_total : 269250 time to create 1 rle with new method : 0.68007493019104 time for calcul the mask position with numpy : 0.04308056831359863 nb_pixel_total : 351413 time to create 1 rle with new method : 0.9663527011871338 time for calcul the mask position with numpy : 0.04524827003479004 nb_pixel_total : 3890 time to create 1 rle with old method : 0.00628972053527832 time for calcul the mask position with numpy : 0.04831242561340332 nb_pixel_total : 135732 time to create 1 rle with old method : 0.1542978286743164 time for calcul the mask position with numpy : 0.044342041015625 nb_pixel_total : 38821 time to create 1 rle with old method : 0.050405263900756836 time for calcul the mask position with numpy : 0.04074573516845703 nb_pixel_total : 28017 time to create 1 rle with old method : 0.03286027908325195 time for calcul the mask position with numpy : 0.03772306442260742 nb_pixel_total : 4998 time to create 1 rle with old method : 0.0057621002197265625 time for calcul the mask position with numpy : 0.03558826446533203 nb_pixel_total : 27881 time to create 1 rle with old method : 0.031055212020874023 time for calcul the mask position with numpy : 0.03275251388549805 nb_pixel_total : 95258 time to create 1 rle with old method : 0.10577249526977539 create new chi : 5.981207609176636 time to delete rle : 0.001996755599975586 batch 1 Loaded 22 chid ids of type : 3726 Number RLEs to save : 14476 TO DO : save crop sub photo not yet done ! save time : 1.5436749458312988 nb_obj : 18 nb_hashtags : 6 time to prepare the origin masks : 16.457030773162842 time for calcul the mask position with numpy : 0.5037562847137451 nb_pixel_total : 6469250 time to create 1 rle with new method : 0.8914926052093506 time for calcul the mask position with numpy : 0.029118061065673828 nb_pixel_total : 28833 time to create 1 rle with old method : 0.03113865852355957 time for calcul the mask position with numpy : 0.040621042251586914 nb_pixel_total : 67685 time to create 1 rle with old method : 0.07303547859191895 time for calcul the mask position with numpy : 0.036184072494506836 nb_pixel_total : 1803 time to create 1 rle with old method : 0.0021851062774658203 time for calcul the mask position with numpy : 0.021126270294189453 nb_pixel_total : 3454 time to create 1 rle with old method : 0.004128932952880859 time for calcul the mask position with numpy : 0.021358013153076172 nb_pixel_total : 2753 time to create 1 rle with old method : 0.003117799758911133 time for calcul the mask position with numpy : 0.02254772186279297 nb_pixel_total : 126663 time to create 1 rle with old method : 0.14002251625061035 time for calcul the mask position with numpy : 0.024013280868530273 nb_pixel_total : 18765 time to create 1 rle with old method : 0.020910978317260742 time for calcul the mask position with numpy : 0.023494243621826172 nb_pixel_total : 8069 time to create 1 rle with old method : 0.009451627731323242 time for calcul the mask position with numpy : 0.02218794822692871 nb_pixel_total : 2486 time to create 1 rle with old method : 0.002880573272705078 time for calcul the mask position with numpy : 0.021065711975097656 nb_pixel_total : 55988 time to create 1 rle with old method : 0.06329703330993652 time for calcul the mask position with numpy : 0.022364377975463867 nb_pixel_total : 113851 time to create 1 rle with old method : 0.12475395202636719 time for calcul the mask position with numpy : 0.021160125732421875 nb_pixel_total : 49415 time to create 1 rle with old method : 0.05565023422241211 time for calcul the mask position with numpy : 0.02322554588317871 nb_pixel_total : 51363 time to create 1 rle with old method : 0.05903220176696777 time for calcul the mask position with numpy : 0.02270340919494629 nb_pixel_total : 18400 time to create 1 rle with old method : 0.020503759384155273 time for calcul the mask position with numpy : 0.022197246551513672 nb_pixel_total : 6084 time to create 1 rle with old method : 0.007443904876708984 time for calcul the mask position with numpy : 0.02236771583557129 nb_pixel_total : 16926 time to create 1 rle with old method : 0.018502235412597656 time for calcul the mask position with numpy : 0.02120375633239746 nb_pixel_total : 5750 time to create 1 rle with old method : 0.006605386734008789 time for calcul the mask position with numpy : 0.021254777908325195 nb_pixel_total : 2702 time to create 1 rle with old method : 0.003155946731567383 create new chi : 2.5229849815368652 time to delete rle : 0.001018524169921875 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 10510 TO DO : save crop sub photo not yet done ! save time : 0.620908260345459 nb_obj : 7 nb_hashtags : 4 time to prepare the origin masks : 6.048711776733398 time for calcul the mask position with numpy : 1.035278558731079 nb_pixel_total : 6609156 time to create 1 rle with new method : 1.6284260749816895 time for calcul the mask position with numpy : 0.02167367935180664 nb_pixel_total : 9248 time to create 1 rle with old method : 0.010721445083618164 time for calcul the mask position with numpy : 0.020941734313964844 nb_pixel_total : 1662 time to create 1 rle with old method : 0.002026081085205078 time for calcul the mask position with numpy : 0.02268671989440918 nb_pixel_total : 188281 time to create 1 rle with new method : 0.8325769901275635 time for calcul the mask position with numpy : 0.036495208740234375 nb_pixel_total : 8287 time to create 1 rle with old method : 0.009002208709716797 time for calcul the mask position with numpy : 0.035192012786865234 nb_pixel_total : 31565 time to create 1 rle with old method : 0.03493952751159668 time for calcul the mask position with numpy : 0.03681588172912598 nb_pixel_total : 161942 time to create 1 rle with new method : 0.4422740936279297 time for calcul the mask position with numpy : 0.032998085021972656 nb_pixel_total : 40099 time to create 1 rle with old method : 0.04394793510437012 create new chi : 4.3377556800842285 time to delete rle : 0.0009131431579589844 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6330 TO DO : save crop sub photo not yet done ! save time : 0.4038562774658203 nb_obj : 23 nb_hashtags : 7 time to prepare the origin masks : 17.51634430885315 time for calcul the mask position with numpy : 0.47173571586608887 nb_pixel_total : 6480370 time to create 1 rle with new method : 0.9546267986297607 time for calcul the mask position with numpy : 0.06278395652770996 nb_pixel_total : 36315 time to create 1 rle with old method : 0.03928995132446289 time for calcul the mask position with numpy : 0.03220033645629883 nb_pixel_total : 31150 time to create 1 rle with old method : 0.03266096115112305 time for calcul the mask position with numpy : 0.0230560302734375 nb_pixel_total : 36859 time to create 1 rle with old method : 0.03946661949157715 time for calcul the mask position with numpy : 0.027458667755126953 nb_pixel_total : 7064 time to create 1 rle with old method : 0.00748896598815918 time for calcul the mask position with numpy : 0.023334980010986328 nb_pixel_total : 7926 time to create 1 rle with old method : 0.008959054946899414 time for calcul the mask position with numpy : 0.028756141662597656 nb_pixel_total : 38202 time to create 1 rle with old method : 0.04283571243286133 time for calcul the mask position with numpy : 0.022048473358154297 nb_pixel_total : 12695 time to create 1 rle with old method : 0.01362156867980957 time for calcul the mask position with numpy : 0.025365114212036133 nb_pixel_total : 14009 time to create 1 rle with old method : 0.015119075775146484 time for calcul the mask position with numpy : 0.022624492645263672 nb_pixel_total : 8521 time to create 1 rle with old method : 0.009134531021118164 time for calcul the mask position with numpy : 0.022198915481567383 nb_pixel_total : 19775 time to create 1 rle with old method : 0.021094799041748047 time for calcul the mask position with numpy : 0.023691892623901367 nb_pixel_total : 2266 time to create 1 rle with old method : 0.002511739730834961 time for calcul the mask position with numpy : 0.02124929428100586 nb_pixel_total : 24875 time to create 1 rle with old method : 0.02929854393005371 time for calcul the mask position with numpy : 0.022954225540161133 nb_pixel_total : 5057 time to create 1 rle with old method : 0.005499362945556641 time for calcul the mask position with numpy : 0.022077083587646484 nb_pixel_total : 2170 time to create 1 rle with old method : 0.0025281906127929688 time for calcul the mask position with numpy : 0.024915218353271484 nb_pixel_total : 5861 time to create 1 rle with old method : 0.006951808929443359 time for calcul the mask position with numpy : 0.025165557861328125 nb_pixel_total : 29228 time to create 1 rle with old method : 0.03880572319030762 time for calcul the mask position with numpy : 0.02958393096923828 nb_pixel_total : 184789 time to create 1 rle with new method : 2.2230935096740723 time for calcul the mask position with numpy : 0.02265620231628418 nb_pixel_total : 8250 time to create 1 rle with old method : 0.00917506217956543 time for calcul the mask position with numpy : 0.0222470760345459 nb_pixel_total : 3835 time to create 1 rle with old method : 0.0042879581451416016 time for calcul the mask position with numpy : 0.021484375 nb_pixel_total : 23786 time to create 1 rle with old method : 0.026108503341674805 time for calcul the mask position with numpy : 0.022520065307617188 nb_pixel_total : 15813 time to create 1 rle with old method : 0.01721787452697754 time for calcul the mask position with numpy : 0.022774457931518555 nb_pixel_total : 19362 time to create 1 rle with old method : 0.02138686180114746 time for calcul the mask position with numpy : 0.022912025451660156 nb_pixel_total : 32062 time to create 1 rle with old method : 0.03662371635437012 create new chi : 4.772422552108765 time to delete rle : 0.001171112060546875 batch 1 Loaded 24 chid ids of type : 3726 Number RLEs to save : 11130 TO DO : save crop sub photo not yet done ! save time : 1.121168851852417 nb_obj : 16 nb_hashtags : 5 time to prepare the origin masks : 21.176023960113525 time for calcul the mask position with numpy : 0.5159945487976074 nb_pixel_total : 5048007 time to create 1 rle with new method : 1.469329595565796 time for calcul the mask position with numpy : 0.025536298751831055 nb_pixel_total : 52199 time to create 1 rle with old method : 0.06387925148010254 time for calcul the mask position with numpy : 0.02659440040588379 nb_pixel_total : 81809 time to create 1 rle with old method : 0.0884859561920166 time for calcul the mask position with numpy : 0.026904821395874023 nb_pixel_total : 100548 time to create 1 rle with old method : 0.11161208152770996 time for calcul the mask position with numpy : 0.02666330337524414 nb_pixel_total : 71686 time to create 1 rle with old method : 0.1155087947845459 time for calcul the mask position with numpy : 0.030742883682250977 nb_pixel_total : 7644 time to create 1 rle with old method : 0.01340484619140625 time for calcul the mask position with numpy : 0.028943300247192383 nb_pixel_total : 10768 time to create 1 rle with old method : 0.01535177230834961 time for calcul the mask position with numpy : 0.0303955078125 nb_pixel_total : 3077 time to create 1 rle with old method : 0.0035619735717773438 time for calcul the mask position with numpy : 0.028269052505493164 nb_pixel_total : 37306 time to create 1 rle with old method : 0.03997015953063965 time for calcul the mask position with numpy : 0.031180143356323242 nb_pixel_total : 240483 time to create 1 rle with new method : 1.086355209350586 time for calcul the mask position with numpy : 0.05203986167907715 nb_pixel_total : 2770 time to create 1 rle with old method : 0.003930568695068359 time for calcul the mask position with numpy : 0.045768022537231445 nb_pixel_total : 107759 time to create 1 rle with old method : 0.11554837226867676 time for calcul the mask position with numpy : 0.04084968566894531 nb_pixel_total : 12128 time to create 1 rle with old method : 0.01397562026977539 time for calcul the mask position with numpy : 0.04350996017456055 nb_pixel_total : 144422 time to create 1 rle with old method : 0.15485620498657227 time for calcul the mask position with numpy : 0.07284402847290039 nb_pixel_total : 996088 time to create 1 rle with new method : 0.8607988357543945 time for calcul the mask position with numpy : 0.04250383377075195 nb_pixel_total : 10168 time to create 1 rle with old method : 0.011497974395751953 time for calcul the mask position with numpy : 0.039557695388793945 nb_pixel_total : 123378 time to create 1 rle with old method : 0.15883326530456543 create new chi : 5.545788288116455 time to delete rle : 0.0026102066040039062 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 17795 TO DO : save crop sub photo not yet done ! save time : 1.0917198657989502 nb_obj : 14 nb_hashtags : 4 time to prepare the origin masks : 19.80192494392395 time for calcul the mask position with numpy : 1.0397861003875732 nb_pixel_total : 5305792 time to create 1 rle with new method : 1.7140061855316162 time for calcul the mask position with numpy : 0.03212285041809082 nb_pixel_total : 53475 time to create 1 rle with old method : 0.05897665023803711 time for calcul the mask position with numpy : 0.05727720260620117 nb_pixel_total : 96464 time to create 1 rle with old method : 0.10855364799499512 time for calcul the mask position with numpy : 0.04278755187988281 nb_pixel_total : 1152 time to create 1 rle with old method : 0.0012695789337158203 time for calcul the mask position with numpy : 0.03307223320007324 nb_pixel_total : 59131 time to create 1 rle with old method : 0.061797142028808594 time for calcul the mask position with numpy : 0.044847965240478516 nb_pixel_total : 355513 time to create 1 rle with new method : 0.8705549240112305 time for calcul the mask position with numpy : 0.023466110229492188 nb_pixel_total : 50112 time to create 1 rle with old method : 0.056783437728881836 time for calcul the mask position with numpy : 0.023073196411132812 nb_pixel_total : 3875 time to create 1 rle with old method : 0.0041849613189697266 time for calcul the mask position with numpy : 0.02340531349182129 nb_pixel_total : 11551 time to create 1 rle with old method : 0.012916088104248047 time for calcul the mask position with numpy : 0.02751779556274414 nb_pixel_total : 247116 time to create 1 rle with new method : 2.154791831970215 time for calcul the mask position with numpy : 0.02478337287902832 nb_pixel_total : 1249 time to create 1 rle with old method : 0.0013320446014404297 time for calcul the mask position with numpy : 0.03405046463012695 nb_pixel_total : 703629 time to create 1 rle with new method : 1.385540246963501 time for calcul the mask position with numpy : 0.04005122184753418 nb_pixel_total : 11206 time to create 1 rle with old method : 0.012104272842407227 time for calcul the mask position with numpy : 0.04620790481567383 nb_pixel_total : 68022 time to create 1 rle with old method : 0.20595431327819824 time for calcul the mask position with numpy : 0.04108691215515137 nb_pixel_total : 81953 time to create 1 rle with old method : 0.08455395698547363 create new chi : 8.403043746948242 time to delete rle : 0.0018076896667480469 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 16533 TO DO : save crop sub photo not yet done ! save time : 1.273172378540039 map_output_result : {1341266663: (0.0, 'Should be the crop_list due to order', 0), 1341266213: (0.0, 'Should be the crop_list due to order', 0), 1341266207: (0.0, 'Should be the crop_list due to order', 0), 1341266194: (0.0, 'Should be the crop_list due to order', 0), 1341266186: (0.0, 'Should be the crop_list due to order', 0), 1341265999: (0.0, 'Should be the crop_list due to order', 0), 1341265987: (0.0, 'Should be the crop_list due to order', 0), 1341265981: (0.0, 'Should be the crop_list due to order', 0), 1341265975: (0.0, 'Should be the crop_list due to order', 0), 1341265971: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987, 1341265981, 1341265975, 1341265971] Looping around the photos to save general results len do output : 10 /1341266663.Didn't retrieve data . /1341266213.Didn't retrieve data . /1341266207.Didn't retrieve data . /1341266194.Didn't retrieve data . /1341266186.Didn't retrieve data . /1341265999.Didn't retrieve data . /1341265987.Didn't retrieve data . /1341265981.Didn't retrieve data . /1341265975.Didn't retrieve data . /1341265971.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266663', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266213', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266207', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266194', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266186', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265999', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265987', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265981', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265975', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265971', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014812946319580078 save_final save missing photos in datou_result : time spend for datou_step_exec : 203.1243863105774 time spend to save output : 0.019263029098510742 total time spend for step 5 : 203.1436493396759 step6:crop_condition Fri Mar 7 00:18:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 139 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741303123_116951 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.81040358543396 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741303128_116951 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.82423734664917 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741303134_116951 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.3929104804992676 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741303140_116951 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7317535877227783 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741303144_116951 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.291203260421753 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741303148_116951 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.1529598236083984 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741303151_116951 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.2061891555786133 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987, 1341265981, 1341265975, 1341265971] Looping around the photos to save general results len do output : 19 /1341338582Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338583Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338584Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338585Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338600Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338601Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338602Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338603Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338604Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338607Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338608Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338617Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338619Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338620Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338621Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338637Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338638Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338639Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338642Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266663', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266213', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266207', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266194', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266186', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265999', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265987', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265981', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265975', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265971', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 67 time used for this insertion : 0.030768394470214844 save_final save missing photos in datou_result : time spend for datou_step_exec : 32.44935989379883 time spend to save output : 0.03175687789916992 total time spend for step 6 : 32.481116771698 step7:ventilate_hashtags_in_portfolio Fri Mar 7 00:19:13 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 21131115 get user id for portfolio 21131115 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131115 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('mal_croppe','cartonnette','kraft','Teint_Dans_La_Masse','Carton_brun','metal','plastique','environnement','Carton_gris','autre_refus','papier','flou')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131115 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('mal_croppe','cartonnette','kraft','Teint_Dans_La_Masse','Carton_brun','metal','plastique','environnement','Carton_gris','autre_refus','papier','flou')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131115 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('mal_croppe','cartonnette','kraft','Teint_Dans_La_Masse','Carton_brun','metal','plastique','environnement','Carton_gris','autre_refus','papier','flou')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/21131189,21131190,21131191,21131192,21131193,21131194,21131195,21131196,21131197,21131198,21131199,21131200?tags=mal_croppe,cartonnette,kraft,Teint_Dans_La_Masse,Carton_brun,metal,plastique,environnement,Carton_gris,autre_refus,papier,flou Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987, 1341265981, 1341265975, 1341265971] Looping around the photos to save general results len do output : 1 /21131115. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266663', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266213', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266207', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266194', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266186', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265999', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265987', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265981', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265975', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265971', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013595104217529297 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.7114334106445312 time spend to save output : 0.014052629470825195 total time spend for step 7 : 1.7254860401153564 step8:final Fri Mar 7 00:19:14 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1341266663: ('0.0628226615711991',), 1341266213: ('0.0628226615711991',), 1341266207: ('0.0628226615711991',), 1341266194: ('0.0628226615711991',), 1341266186: ('0.0628226615711991',), 1341265999: ('0.0628226615711991',), 1341265987: ('0.0628226615711991',), 1341265981: ('0.0628226615711991',), 1341265975: ('0.0628226615711991',), 1341265971: ('0.0628226615711991',)} new output for save of step final : {1341266663: ('0.0628226615711991',), 1341266213: ('0.0628226615711991',), 1341266207: ('0.0628226615711991',), 1341266194: ('0.0628226615711991',), 1341266186: ('0.0628226615711991',), 1341265999: ('0.0628226615711991',), 1341265987: ('0.0628226615711991',), 1341265981: ('0.0628226615711991',), 1341265975: ('0.0628226615711991',), 1341265971: ('0.0628226615711991',)} [1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987, 1341265981, 1341265975, 1341265971] Looping around the photos to save general results len do output : 10 /1341266663.Didn't retrieve data . /1341266213.Didn't retrieve data . /1341266207.Didn't retrieve data . /1341266194.Didn't retrieve data . /1341266186.Didn't retrieve data . /1341265999.Didn't retrieve data . /1341265987.Didn't retrieve data . /1341265981.Didn't retrieve data . /1341265975.Didn't retrieve data . /1341265971.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266663', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266213', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266207', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266194', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266186', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265999', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265987', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265981', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265975', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265971', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013405323028564453 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.1149907112121582 time spend to save output : 0.014414072036743164 total time spend for step 8 : 0.12940478324890137 step9:velours_tree Fri Mar 7 00:19:15 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.25444722175598145 time spend to save output : 5.4836273193359375e-05 total time spend for step 9 : 0.2545020580291748 step10:send_mail_cod Fri Mar 7 00:19:15 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21131115_07-03-2025_00_19_15.pdf 21131189 imagette211311891741303155 21131190 change filename to text .imagette211311901741303155 21131191 change filename to text .imagette211311911741303155 21131194 imagette211311941741303155 21131195 change filename to text .change filename to text .change filename to text .imagette211311951741303155 21131198 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211311981741303155 21131200 imagette211312001741303156 21131192 change filename to text .change filename to text .change filename to text .change filename to text .imagette211311921741303156 21131193 change filename to text .change filename to text .change filename to text .imagette211311931741303156 21131197 change filename to text .change filename to text .imagette211311971741303157 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21131115 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21131189,21131190,21131191,21131192,21131193,21131194,21131195,21131196,21131197,21131198,21131199,21131200?tags=mal_croppe,cartonnette,kraft,Teint_Dans_La_Masse,Carton_brun,metal,plastique,environnement,Carton_gris,autre_refus,papier,flou your option no_mail is active, we will not send the real mail to your client args[1341266663] : ('0.0628226615711991',) no score found for photo 1341266663 We are sending mail with results at report@fotonower.com args[1341266213] : ('0.0628226615711991',) no score found for photo 1341266213 We are sending mail with results at report@fotonower.com args[1341266207] : ('0.0628226615711991',) no score found for photo 1341266207 We are sending mail with results at report@fotonower.com args[1341266194] : ('0.0628226615711991',) no score found for photo 1341266194 We are sending mail with results at report@fotonower.com args[1341266186] : ('0.0628226615711991',) no score found for photo 1341266186 We are sending mail with results at report@fotonower.com args[1341265999] : ('0.0628226615711991',) no score found for photo 1341265999 We are sending mail with results at report@fotonower.com args[1341265987] : ('0.0628226615711991',) no score found for photo 1341265987 We are sending mail with results at report@fotonower.com args[1341265981] : ('0.0628226615711991',) no score found for photo 1341265981 We are sending mail with results at report@fotonower.com args[1341265975] : ('0.0628226615711991',) no score found for photo 1341265975 We are sending mail with results at report@fotonower.com args[1341265971] : ('0.0628226615711991',) no score found for photo 1341265971 We are sending mail with results at report@fotonower.com refus_total : 0.0628226615711991 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=21131115 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1341265562,1341265574,1341265629,1341265698,1341265754,1341266213,1341265971,1341265555,1341265981,1341265987,1341265999,1341266186,1341265975,1341266194,1341266207,1341265967,1341266663) Found this number of photos: 17 begin to download photo : 1341265562 begin to download photo : 1341265754 begin to download photo : 1341265981 begin to download photo : 1341265975 begin to download photo : 1341266663 download finish for photo 1341265562 begin to download photo : 1341265574 download finish for photo 1341265975 begin to download photo : 1341266194 download finish for photo 1341265754 begin to download photo : 1341266213 download finish for photo 1341265981 begin to download photo : 1341265987 download finish for photo 1341266663 download finish for photo 1341266194 begin to download photo : 1341266207 download finish for photo 1341265987 begin to download photo : 1341265999 download finish for photo 1341265574 begin to download photo : 1341265629 download finish for photo 1341266213 begin to download photo : 1341265971 download finish for photo 1341266207 begin to download photo : 1341265967 download finish for photo 1341265999 begin to download photo : 1341266186 download finish for photo 1341265629 begin to download photo : 1341265698 download finish for photo 1341265971 begin to download photo : 1341265555 download finish for photo 1341265967 download finish for photo 1341266186 download finish for photo 1341265555 download finish for photo 1341265698 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131115_07-03-2025_00_19_15.pdf results_Auto_P21131115_07-03-2025_00_19_15.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131115_07-03-2025_00_19_15.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','21131115','results_Auto_P21131115_07-03-2025_00_19_15.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131115_07-03-2025_00_19_15.pdf','pdf','','0.54','0.0628226615711991') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987, 1341265981, 1341265975, 1341265971] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266663', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266213', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266207', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266194', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266186', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265999', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265987', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265981', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265975', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265971', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.014159440994262695 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.860484600067139 time spend to save output : 0.01465296745300293 total time spend for step 10 : 6.875137567520142 step11:split_time_score Fri Mar 7 00:19:22 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('17', 17),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 06032025 21131115 Nombre de photos uploadées : 17 / 23040 (0%) 06032025 21131115 Nombre de photos taguées (types de déchets): 0 / 17 (0%) 06032025 21131115 Nombre de photos taguées (volume) : 0 / 17 (0%) elapsed_time : load_data_split_time_score 2.384185791015625e-06 elapsed_time : order_list_meta_photo_and_scores 1.1682510375976562e-05 ????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0012364387512207031 elapsed_time : insert_dashboard_record_day_entry 0.023529529571533203 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131086 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131087 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131088 order by id desc limit 1 Qualite : 0.08488825436881216 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21110324_06-03-2025_13_24_09.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21110324 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21110324 AND mptpi.`type`=3327 To do Qualite : 0.08456570303462264 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21110325_06-03-2025_09_27_50.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21110325 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21110325 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21110326 order by id desc limit 1 Qualite : 0.05850682993931322 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21110327_06-03-2025_09_32_24.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21110327 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21110327 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131089 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131090 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131091 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131092 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131093 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131094 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131095 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131096 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131097 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131098 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131099 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131100 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131101 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131102 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131103 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131104 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131105 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131106 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131107 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131108 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131109 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131110 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131111 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131112 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131113 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131114 order by id desc limit 1 Qualite : 0.0628226615711991 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131115_07-03-2025_00_19_15.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131115 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131115 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'06032025': {'nb_upload': 17, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987, 1341265981, 1341265975, 1341265971] Looping around the photos to save general results len do output : 1 /21131115Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266663', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266213', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266207', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266194', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341266186', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265999', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265987', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265981', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265975', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', '21131115', '1341265971', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.016079187393188477 save_final save missing photos in datou_result : time spend for datou_step_exec : 24.043474912643433 time spend to save output : 0.016305208206176758 total time spend for step 11 : 24.05978012084961 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 137.81user 241.11system 7:24.14elapsed 85%CPU (0avgtext+0avgdata 4260272maxresident)k 814976inputs+96400outputs (27259major+12334313minor)pagefaults 0swaps