python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 95702 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 26>10 please execute split_portfolio.py -i 20920571 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2623036'] with mtr_portfolio_ids : ['20920571'] and first list_photo_ids : [] new path : /proc/95702/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.184126853942871 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 28 10:08:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5304 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-28 10:08:33.155590: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-28 10:08:33.183354: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-28 10:08:33.185142: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f4d1c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-28 10:08:33.185169: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-28 10:08:33.188485: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-28 10:08:33.456994: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x59b3bf0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-28 10:08:33.457063: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-28 10:08:33.458064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 10:08:33.458737: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 10:08:33.463547: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 10:08:33.466944: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 10:08:33.468434: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 10:08:33.472136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 10:08:33.474032: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 10:08:33.481942: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 10:08:33.483860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 10:08:33.484048: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 10:08:33.485043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-28 10:08:33.485070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-28 10:08:33.485084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-28 10:08:33.486767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4843 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-28 10:08:33.823096: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 10:08:33.823212: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 10:08:33.823234: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 10:08:33.823254: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 10:08:33.823273: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 10:08:33.823291: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 10:08:33.823310: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 10:08:33.823329: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 10:08:33.826293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 10:08:33.827657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 10:08:33.827716: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 10:08:33.827737: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 10:08:33.827756: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 10:08:33.827773: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 10:08:33.827792: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 10:08:33.827809: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 10:08:33.827828: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 10:08:33.829033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 10:08:33.829064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-28 10:08:33.829074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-28 10:08:33.829084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-28 10:08:33.830325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4843 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-28 10:08:42.058157: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 10:08:42.245983: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 13 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 9 Detection mask done ! Trying to reset tf kernel 96153 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 15 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5304 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.29892492294311523 nb_pixel_total : 944256 time to create 1 rle with new method : 0.09698915481567383 length of segment : 2338 time for calcul the mask position with numpy : 0.0038034915924072266 nb_pixel_total : 246314 time to create 1 rle with new method : 0.01083683967590332 length of segment : 828 time for calcul the mask position with numpy : 0.00031638145446777344 nb_pixel_total : 19446 time to create 1 rle with old method : 0.020894765853881836 length of segment : 163 time for calcul the mask position with numpy : 0.001971006393432617 nb_pixel_total : 108878 time to create 1 rle with old method : 0.11507010459899902 length of segment : 859 time for calcul the mask position with numpy : 0.0005209445953369141 nb_pixel_total : 34322 time to create 1 rle with old method : 0.04283452033996582 length of segment : 183 time for calcul the mask position with numpy : 0.0002567768096923828 nb_pixel_total : 14699 time to create 1 rle with old method : 0.016327381134033203 length of segment : 192 time for calcul the mask position with numpy : 0.0005671977996826172 nb_pixel_total : 50592 time to create 1 rle with old method : 0.05543327331542969 length of segment : 208 time for calcul the mask position with numpy : 0.005647182464599609 nb_pixel_total : 314481 time to create 1 rle with new method : 0.01372075080871582 length of segment : 1244 time for calcul the mask position with numpy : 6.198883056640625e-05 nb_pixel_total : 760 time to create 1 rle with old method : 0.0009472370147705078 length of segment : 34 time for calcul the mask position with numpy : 0.00028204917907714844 nb_pixel_total : 8019 time to create 1 rle with old method : 0.008748769760131836 length of segment : 108 time for calcul the mask position with numpy : 0.006569862365722656 nb_pixel_total : 286905 time to create 1 rle with new method : 0.017040014266967773 length of segment : 1655 time for calcul the mask position with numpy : 0.001882314682006836 nb_pixel_total : 81442 time to create 1 rle with old method : 0.0831754207611084 length of segment : 585 time for calcul the mask position with numpy : 0.004396200180053711 nb_pixel_total : 341736 time to create 1 rle with new method : 0.011391162872314453 length of segment : 1032 time for calcul the mask position with numpy : 0.00124359130859375 nb_pixel_total : 62486 time to create 1 rle with old method : 0.06635546684265137 length of segment : 457 time for calcul the mask position with numpy : 0.011394739151000977 nb_pixel_total : 607701 time to create 1 rle with new method : 0.05559730529785156 length of segment : 1136 time for calcul the mask position with numpy : 0.0010344982147216797 nb_pixel_total : 37729 time to create 1 rle with old method : 0.043024301528930664 length of segment : 260 time for calcul the mask position with numpy : 0.0004439353942871094 nb_pixel_total : 12638 time to create 1 rle with old method : 0.014033317565917969 length of segment : 183 time for calcul the mask position with numpy : 0.002229452133178711 nb_pixel_total : 111958 time to create 1 rle with old method : 0.12081384658813477 length of segment : 598 time for calcul the mask position with numpy : 9.608268737792969e-05 nb_pixel_total : 815 time to create 1 rle with old method : 0.0010662078857421875 length of segment : 20 time for calcul the mask position with numpy : 0.0001537799835205078 nb_pixel_total : 8400 time to create 1 rle with old method : 0.009975194931030273 length of segment : 107 time for calcul the mask position with numpy : 0.0004475116729736328 nb_pixel_total : 14262 time to create 1 rle with old method : 0.015949249267578125 length of segment : 205 time for calcul the mask position with numpy : 0.0021224021911621094 nb_pixel_total : 73450 time to create 1 rle with old method : 0.08129620552062988 length of segment : 350 time for calcul the mask position with numpy : 0.00018548965454101562 nb_pixel_total : 2003 time to create 1 rle with old method : 0.002405405044555664 length of segment : 64 time for calcul the mask position with numpy : 0.00013780593872070312 nb_pixel_total : 3774 time to create 1 rle with old method : 0.004655599594116211 length of segment : 94 time for calcul the mask position with numpy : 0.00031566619873046875 nb_pixel_total : 6317 time to create 1 rle with old method : 0.010498762130737305 length of segment : 131 time for calcul the mask position with numpy : 0.000217437744140625 nb_pixel_total : 3947 time to create 1 rle with old method : 0.006725788116455078 length of segment : 83 time for calcul the mask position with numpy : 0.0008916854858398438 nb_pixel_total : 18577 time to create 1 rle with old method : 0.02563190460205078 length of segment : 276 time for calcul the mask position with numpy : 0.009622573852539062 nb_pixel_total : 534179 time to create 1 rle with new method : 0.01770472526550293 length of segment : 1534 time for calcul the mask position with numpy : 0.000934600830078125 nb_pixel_total : 67591 time to create 1 rle with old method : 0.07641172409057617 length of segment : 302 time for calcul the mask position with numpy : 0.015563011169433594 nb_pixel_total : 699806 time to create 1 rle with new method : 0.027262210845947266 length of segment : 906 time for calcul the mask position with numpy : 0.0008411407470703125 nb_pixel_total : 45783 time to create 1 rle with old method : 0.05031991004943848 length of segment : 269 time for calcul the mask position with numpy : 0.000743865966796875 nb_pixel_total : 37889 time to create 1 rle with old method : 0.04292464256286621 length of segment : 185 time for calcul the mask position with numpy : 0.0007312297821044922 nb_pixel_total : 36884 time to create 1 rle with old method : 0.05497169494628906 length of segment : 395 time for calcul the mask position with numpy : 0.003655672073364258 nb_pixel_total : 305142 time to create 1 rle with new method : 0.008060455322265625 length of segment : 906 time for calcul the mask position with numpy : 0.0009047985076904297 nb_pixel_total : 38217 time to create 1 rle with old method : 0.04285717010498047 length of segment : 412 time for calcul the mask position with numpy : 0.0001068115234375 nb_pixel_total : 1176 time to create 1 rle with old method : 0.0014643669128417969 length of segment : 41 time for calcul the mask position with numpy : 0.009007930755615234 nb_pixel_total : 494337 time to create 1 rle with new method : 0.015921831130981445 length of segment : 1064 time for calcul the mask position with numpy : 0.0005314350128173828 nb_pixel_total : 39317 time to create 1 rle with old method : 0.04703164100646973 length of segment : 150 time for calcul the mask position with numpy : 0.0019290447235107422 nb_pixel_total : 103417 time to create 1 rle with old method : 0.11972904205322266 length of segment : 499 time for calcul the mask position with numpy : 0.0004143714904785156 nb_pixel_total : 24741 time to create 1 rle with old method : 0.028041362762451172 length of segment : 182 time for calcul the mask position with numpy : 0.0003199577331542969 nb_pixel_total : 25722 time to create 1 rle with old method : 0.028855562210083008 length of segment : 186 time for calcul the mask position with numpy : 0.0002846717834472656 nb_pixel_total : 22337 time to create 1 rle with old method : 0.025194644927978516 length of segment : 179 time for calcul the mask position with numpy : 0.0005474090576171875 nb_pixel_total : 31231 time to create 1 rle with old method : 0.036623477935791016 length of segment : 130 time for calcul the mask position with numpy : 0.0016667842864990234 nb_pixel_total : 98024 time to create 1 rle with old method : 0.108856201171875 length of segment : 438 time for calcul the mask position with numpy : 0.004813194274902344 nb_pixel_total : 243530 time to create 1 rle with new method : 0.009156227111816406 length of segment : 848 time for calcul the mask position with numpy : 0.0008442401885986328 nb_pixel_total : 32380 time to create 1 rle with old method : 0.03710532188415527 length of segment : 352 time for calcul the mask position with numpy : 0.00042891502380371094 nb_pixel_total : 16430 time to create 1 rle with old method : 0.018980026245117188 length of segment : 186 time for calcul the mask position with numpy : 0.0003750324249267578 nb_pixel_total : 12992 time to create 1 rle with old method : 0.014915227890014648 length of segment : 213 time for calcul the mask position with numpy : 0.00013184547424316406 nb_pixel_total : 5159 time to create 1 rle with old method : 0.006276369094848633 length of segment : 123 time for calcul the mask position with numpy : 0.00014138221740722656 nb_pixel_total : 4532 time to create 1 rle with old method : 0.005535125732421875 length of segment : 131 time for calcul the mask position with numpy : 0.0039789676666259766 nb_pixel_total : 296956 time to create 1 rle with new method : 0.009319782257080078 length of segment : 1048 time for calcul the mask position with numpy : 0.00037384033203125 nb_pixel_total : 20385 time to create 1 rle with old method : 0.022638797760009766 length of segment : 318 time for calcul the mask position with numpy : 0.00028586387634277344 nb_pixel_total : 15662 time to create 1 rle with old method : 0.017603397369384766 length of segment : 175 time for calcul the mask position with numpy : 0.000911712646484375 nb_pixel_total : 42567 time to create 1 rle with old method : 0.04734992980957031 length of segment : 308 time for calcul the mask position with numpy : 0.000255584716796875 nb_pixel_total : 7627 time to create 1 rle with old method : 0.008956432342529297 length of segment : 78 time for calcul the mask position with numpy : 0.003010272979736328 nb_pixel_total : 110699 time to create 1 rle with old method : 0.12223625183105469 length of segment : 671 time for calcul the mask position with numpy : 0.01378321647644043 nb_pixel_total : 849721 time to create 1 rle with new method : 0.03762245178222656 length of segment : 1539 time for calcul the mask position with numpy : 0.0001423358917236328 nb_pixel_total : 5007 time to create 1 rle with old method : 0.006013154983520508 length of segment : 72 time for calcul the mask position with numpy : 0.003596782684326172 nb_pixel_total : 302247 time to create 1 rle with new method : 0.00822305679321289 length of segment : 970 time for calcul the mask position with numpy : 0.0003170967102050781 nb_pixel_total : 15441 time to create 1 rle with old method : 0.01752185821533203 length of segment : 106 time for calcul the mask position with numpy : 0.0011360645294189453 nb_pixel_total : 67473 time to create 1 rle with old method : 0.07653403282165527 length of segment : 395 time for calcul the mask position with numpy : 0.001115560531616211 nb_pixel_total : 50489 time to create 1 rle with old method : 0.05752825736999512 length of segment : 373 time for calcul the mask position with numpy : 0.00021409988403320312 nb_pixel_total : 7031 time to create 1 rle with old method : 0.008059024810791016 length of segment : 80 time for calcul the mask position with numpy : 0.00032258033752441406 nb_pixel_total : 14580 time to create 1 rle with old method : 0.01678466796875 length of segment : 142 time for calcul the mask position with numpy : 0.00016188621520996094 nb_pixel_total : 3627 time to create 1 rle with old method : 0.004286050796508789 length of segment : 71 time for calcul the mask position with numpy : 0.0008046627044677734 nb_pixel_total : 48887 time to create 1 rle with old method : 0.05301189422607422 length of segment : 180 time for calcul the mask position with numpy : 0.0008695125579833984 nb_pixel_total : 35130 time to create 1 rle with old method : 0.038286447525024414 length of segment : 279 time for calcul the mask position with numpy : 0.014404058456420898 nb_pixel_total : 806782 time to create 1 rle with new method : 0.3160438537597656 length of segment : 1213 time for calcul the mask position with numpy : 0.00011420249938964844 nb_pixel_total : 2708 time to create 1 rle with old method : 0.0034186840057373047 length of segment : 96 time for calcul the mask position with numpy : 0.0004761219024658203 nb_pixel_total : 29585 time to create 1 rle with old method : 0.0330815315246582 length of segment : 307 time for calcul the mask position with numpy : 0.00010347366333007812 nb_pixel_total : 2372 time to create 1 rle with old method : 0.0029115676879882812 length of segment : 96 time for calcul the mask position with numpy : 0.0005164146423339844 nb_pixel_total : 30443 time to create 1 rle with old method : 0.03377509117126465 length of segment : 141 time for calcul the mask position with numpy : 0.0038192272186279297 nb_pixel_total : 307284 time to create 1 rle with new method : 0.009434938430786133 length of segment : 934 time for calcul the mask position with numpy : 0.00026297569274902344 nb_pixel_total : 15253 time to create 1 rle with old method : 0.017552614212036133 length of segment : 171 time for calcul the mask position with numpy : 0.0017926692962646484 nb_pixel_total : 104754 time to create 1 rle with old method : 0.11986851692199707 length of segment : 469 time for calcul the mask position with numpy : 0.013612031936645508 nb_pixel_total : 816474 time to create 1 rle with new method : 0.05820941925048828 length of segment : 1169 time for calcul the mask position with numpy : 0.003541231155395508 nb_pixel_total : 304622 time to create 1 rle with new method : 0.007728099822998047 length of segment : 912 time for calcul the mask position with numpy : 0.0002601146697998047 nb_pixel_total : 15392 time to create 1 rle with old method : 0.017716646194458008 length of segment : 78 time for calcul the mask position with numpy : 9.250640869140625e-05 nb_pixel_total : 4239 time to create 1 rle with old method : 0.004972696304321289 length of segment : 79 time for calcul the mask position with numpy : 6.031990051269531e-05 nb_pixel_total : 1937 time to create 1 rle with old method : 0.0023593902587890625 length of segment : 37 time for calcul the mask position with numpy : 0.0013225078582763672 nb_pixel_total : 84884 time to create 1 rle with old method : 0.09765076637268066 length of segment : 424 time for calcul the mask position with numpy : 7.319450378417969e-05 nb_pixel_total : 1859 time to create 1 rle with old method : 0.0022802352905273438 length of segment : 54 time spent for convertir_results : 8.455804347991943 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 82 chid ids of type : 3663 Number RLEs to save : 36009 save missing photos in datou_result : time spend for datou_step_exec : 32.570942640304565 time spend to save output : 2.0678157806396484 total time spend for step 1 : 34.638758420944214 step2:crop_condition Fri Feb 28 10:09:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 82 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340106943, 1340106890, 1340106888, 1340106830, 1340106827, 1340106725, 1340106720, 1340106227, 1340106225, 1340105977] Looping around the photos to save general results len do output : 31 /-3689842787Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842799Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842804Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842798Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842813Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842814Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842820Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842829Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842828Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842826Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842805Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842802Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842842Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842857Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842823Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842854Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842848Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842796Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842811Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842807Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842817Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842831Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842840Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842847Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842852Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842849Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842860Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842845Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842800Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842846Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689842824Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106943', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106890', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106888', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106830', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106827', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106725', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106720', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106227', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106225', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340105977', None, None, None, None, None, '2623036') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 103 time used for this insertion : 0.024186372756958008 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.619021415710449 time spend to save output : 0.02512073516845703 total time spend for step 2 : 6.644142150878906 step3:thcl Fri Feb 28 10:09:11 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.0039958953857421875 time to convert the images to numpy array : 3.337860107421875e-06 time to import caffe and check if the image exist : 0.0040700435638427734 time to convert the images to numpy array : 2.6226043701171875e-06 time to import caffe and check if the image exist : 0.007271289825439453 time to convert the images to numpy array : 0.0173952579498291 time to import caffe and check if the image exist : 0.005911588668823242 time to convert the images to numpy array : 0.02147078514099121 time to import caffe and check if the image exist : 0.004452228546142578 time to convert the images to numpy array : 0.026914358139038086 time to import caffe and check if the image exist : 0.00830078125 time to convert the images to numpy array : 0.02559041976928711 time to import caffe and check if the image exist : 0.006242990493774414 time to convert the images to numpy array : 0.027103424072265625 time to import caffe and check if the image exist : 0.005915403366088867 time to convert the images to numpy array : 0.029586315155029297 time to import caffe and check if the image exist : 0.009565114974975586 time to convert the images to numpy array : 0.026740074157714844 time to import caffe and check if the image exist : 0.00776982307434082 time to convert the images to numpy array : 0.029454469680786133 total time to convert the images to numpy array : 0.2768256664276123 list photo_ids error: [] list photo_ids correct : [-3689842817, -3689842831, -3689842840, -3689842847, -3689842852, -3689842849, -3689842860, -3689842845, -3689842787, -3689842799, -3689842804, -3689842798, -3689842813, -3689842814, -3689842820, -3689842829, -3689842800, -3689842846, -3689842824, -3689842828, -3689842826, -3689842805, -3689842802, -3689842848, -3689842796, -3689842811, -3689842807, -3689842842, -3689842857, -3689842823, -3689842854] number of photos to traite : 31 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5304 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5085 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.23125243186950684 time used to do the prediction : 0.12529468536376953 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.25243067741394043 time used to do the prediction : 0.09444665908813477 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.811981201171875e-05 save missing photos in datou_result : time spend for datou_step_exec : 7.701033353805542 time spend to save output : 0.00034356117248535156 total time spend for step 3 : 7.701376914978027 step4:merge_mask_thcl_custom Fri Feb 28 10:09:19 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 82 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1340106943, 1340106890, 1340106888, 1340106830, 1340106827, 1340106725, 1340106720, 1340106227, 1340106225, 1340105977] Looping around the photos to save general results len do output : 10 /1340106943Didn't retrieve data .Didn't retrieve data . /1340106890Didn't retrieve data .Didn't retrieve data . /1340106888Didn't retrieve data .Didn't retrieve data . /1340106830Didn't retrieve data .Didn't retrieve data . /1340106827Didn't retrieve data .Didn't retrieve data . /1340106725Didn't retrieve data .Didn't retrieve data . /1340106720Didn't retrieve data .Didn't retrieve data . /1340106227Didn't retrieve data .Didn't retrieve data . /1340106225Didn't retrieve data .Didn't retrieve data . /1340105977Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106943', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106890', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106888', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106830', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106827', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106725', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106720', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106227', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106225', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340105977', None, None, None, None, None, '2623036') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014035701751708984 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.06390619277954102 time spend to save output : 0.014428377151489258 total time spend for step 4 : 0.07833456993103027 step5:rle_unique_nms_with_priority Fri Feb 28 10:09:19 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 9 nb_hashtags : 2 time to prepare the origin masks : 1.6563568115234375 time for calcul the mask position with numpy : 0.269869327545166 nb_pixel_total : 984924 time to create 1 rle with new method : 0.09322118759155273 time for calcul the mask position with numpy : 0.0064809322357177734 nb_pixel_total : 750 time to create 1 rle with old method : 0.0008709430694580078 time for calcul the mask position with numpy : 0.007279872894287109 nb_pixel_total : 68061 time to create 1 rle with old method : 0.07975506782531738 time for calcul the mask position with numpy : 0.007284879684448242 nb_pixel_total : 6610 time to create 1 rle with old method : 0.007431745529174805 time for calcul the mask position with numpy : 0.006384134292602539 nb_pixel_total : 11150 time to create 1 rle with old method : 0.012050151824951172 time for calcul the mask position with numpy : 0.006621837615966797 nb_pixel_total : 3167 time to create 1 rle with old method : 0.0034723281860351562 time for calcul the mask position with numpy : 0.006513118743896484 nb_pixel_total : 29311 time to create 1 rle with old method : 0.03169608116149902 time for calcul the mask position with numpy : 0.0066339969635009766 nb_pixel_total : 7869 time to create 1 rle with old method : 0.008556604385375977 time for calcul the mask position with numpy : 0.006227970123291016 nb_pixel_total : 9821 time to create 1 rle with old method : 0.01077723503112793 time for calcul the mask position with numpy : 0.27843475341796875 nb_pixel_total : 951937 time to create 1 rle with new method : 0.08536481857299805 create new chi : 0.953585147857666 time to delete rle : 0.015326261520385742 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 8364 TO DO : save crop sub photo not yet done ! save time : 0.4976840019226074 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 1.2198076248168945 time for calcul the mask position with numpy : 0.04454445838928223 nb_pixel_total : 1637969 time to create 1 rle with new method : 0.046117544174194336 time for calcul the mask position with numpy : 0.010398149490356445 nb_pixel_total : 79310 time to create 1 rle with old method : 0.08841753005981445 time for calcul the mask position with numpy : 0.01807093620300293 nb_pixel_total : 285914 time to create 1 rle with new method : 0.03608059883117676 time for calcul the mask position with numpy : 0.006333589553833008 nb_pixel_total : 7983 time to create 1 rle with old method : 0.008532524108886719 time for calcul the mask position with numpy : 0.0066907405853271484 nb_pixel_total : 62424 time to create 1 rle with old method : 0.06757926940917969 create new chi : 0.3336515426635742 time to delete rle : 0.0006780624389648438 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 6564 TO DO : save crop sub photo not yet done ! save time : 0.3995394706726074 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 0.8650929927825928 time for calcul the mask position with numpy : 0.4319648742675781 nb_pixel_total : 1193943 time to create 1 rle with new method : 0.4152848720550537 time for calcul the mask position with numpy : 0.0064313411712646484 nb_pixel_total : 107160 time to create 1 rle with old method : 0.11610102653503418 time for calcul the mask position with numpy : 0.006300210952758789 nb_pixel_total : 8356 time to create 1 rle with old method : 0.009473562240600586 time for calcul the mask position with numpy : 0.00636601448059082 nb_pixel_total : 1997 time to create 1 rle with old method : 0.0022869110107421875 time for calcul the mask position with numpy : 0.006289958953857422 nb_pixel_total : 10690 time to create 1 rle with old method : 0.012456178665161133 time for calcul the mask position with numpy : 0.006651639938354492 nb_pixel_total : 808 time to create 1 rle with old method : 0.000919342041015625 time for calcul the mask position with numpy : 0.006853580474853516 nb_pixel_total : 11899 time to create 1 rle with old method : 0.013782739639282227 time for calcul the mask position with numpy : 0.015228748321533203 nb_pixel_total : 627791 time to create 1 rle with new method : 0.196946382522583 time for calcul the mask position with numpy : 0.008270740509033203 nb_pixel_total : 37665 time to create 1 rle with old method : 0.04227280616760254 time for calcul the mask position with numpy : 0.0074803829193115234 nb_pixel_total : 73291 time to create 1 rle with old method : 0.08201837539672852 create new chi : 1.403221607208252 time to delete rle : 0.0009200572967529297 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 6298 TO DO : save crop sub photo not yet done ! save time : 0.37488770484924316 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 0.49317502975463867 time for calcul the mask position with numpy : 0.015926837921142578 nb_pixel_total : 1442802 time to create 1 rle with new method : 0.0320897102355957 time for calcul the mask position with numpy : 0.006491661071777344 nb_pixel_total : 6295 time to create 1 rle with old method : 0.0069730281829833984 time for calcul the mask position with numpy : 0.0065724849700927734 nb_pixel_total : 67391 time to create 1 rle with old method : 0.0732572078704834 time for calcul the mask position with numpy : 0.01069331169128418 nb_pixel_total : 534659 time to create 1 rle with new method : 0.030883073806762695 time for calcul the mask position with numpy : 0.006539821624755859 nb_pixel_total : 18512 time to create 1 rle with old method : 0.02026677131652832 time for calcul the mask position with numpy : 0.006037473678588867 nb_pixel_total : 3941 time to create 1 rle with old method : 0.004453897476196289 create new chi : 0.22805094718933105 time to delete rle : 0.0005753040313720703 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 5500 TO DO : save crop sub photo not yet done ! save time : 0.3667948246002197 nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 1.396280288696289 time for calcul the mask position with numpy : 0.3251039981842041 nb_pixel_total : 1150173 time to create 1 rle with new method : 0.10915470123291016 time for calcul the mask position with numpy : 0.00714874267578125 nb_pixel_total : 1165 time to create 1 rle with old method : 0.0013988018035888672 time for calcul the mask position with numpy : 0.007630825042724609 nb_pixel_total : 45603 time to create 1 rle with old method : 0.051666259765625 time for calcul the mask position with numpy : 0.008007526397705078 nb_pixel_total : 96864 time to create 1 rle with old method : 0.1067969799041748 time for calcul the mask position with numpy : 0.0073277950286865234 nb_pixel_total : 5623 time to create 1 rle with old method : 0.0063359737396240234 time for calcul the mask position with numpy : 0.0076406002044677734 nb_pixel_total : 36678 time to create 1 rle with old method : 0.03940224647521973 time for calcul the mask position with numpy : 0.006944894790649414 nb_pixel_total : 37700 time to create 1 rle with old method : 0.040109872817993164 time for calcul the mask position with numpy : 0.013438701629638672 nb_pixel_total : 699794 time to create 1 rle with new method : 0.10541319847106934 create new chi : 0.8635711669921875 time to delete rle : 0.0006344318389892578 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 5831 TO DO : save crop sub photo not yet done ! save time : 0.3603043556213379 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 0.40003156661987305 time for calcul the mask position with numpy : 0.020718097686767578 nb_pixel_total : 1411829 time to create 1 rle with new method : 0.0995783805847168 time for calcul the mask position with numpy : 0.0067517757415771484 nb_pixel_total : 76 time to create 1 rle with old method : 0.00018310546875 time for calcul the mask position with numpy : 0.006935834884643555 nb_pixel_total : 103380 time to create 1 rle with old method : 0.10383129119873047 time for calcul the mask position with numpy : 0.007138490676879883 nb_pixel_total : 38136 time to create 1 rle with old method : 0.040715694427490234 time for calcul the mask position with numpy : 0.007355451583862305 nb_pixel_total : 25623 time to create 1 rle with old method : 0.026432037353515625 time for calcul the mask position with numpy : 0.01303243637084961 nb_pixel_total : 494556 time to create 1 rle with new method : 0.0650327205657959 create new chi : 0.406818151473999 time to delete rle : 0.0005445480346679688 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 4887 TO DO : save crop sub photo not yet done ! save time : 0.295367956161499 nb_obj : 10 nb_hashtags : 3 time to prepare the origin masks : 1.446666955947876 time for calcul the mask position with numpy : 0.04332137107849121 nb_pixel_total : 1311297 time to create 1 rle with new method : 0.24555492401123047 time for calcul the mask position with numpy : 0.007175922393798828 nb_pixel_total : 14669 time to create 1 rle with old method : 0.015813827514648438 time for calcul the mask position with numpy : 0.006521701812744141 nb_pixel_total : 20128 time to create 1 rle with old method : 0.02199530601501465 time for calcul the mask position with numpy : 0.008584260940551758 nb_pixel_total : 289807 time to create 1 rle with new method : 0.08963847160339355 time for calcul the mask position with numpy : 0.006472110748291016 nb_pixel_total : 16356 time to create 1 rle with old method : 0.018074989318847656 time for calcul the mask position with numpy : 0.006459951400756836 nb_pixel_total : 12883 time to create 1 rle with old method : 0.014354467391967773 time for calcul the mask position with numpy : 0.006768941879272461 nb_pixel_total : 98007 time to create 1 rle with old method : 0.10905575752258301 time for calcul the mask position with numpy : 0.0065882205963134766 nb_pixel_total : 4736 time to create 1 rle with old method : 0.005314350128173828 time for calcul the mask position with numpy : 0.007668495178222656 nb_pixel_total : 242290 time to create 1 rle with new method : 0.030750751495361328 time for calcul the mask position with numpy : 0.0073244571685791016 nb_pixel_total : 32262 time to create 1 rle with old method : 0.035218000411987305 time for calcul the mask position with numpy : 0.006327629089355469 nb_pixel_total : 31165 time to create 1 rle with old method : 0.03396487236022949 create new chi : 0.7449479103088379 time to delete rle : 0.0009243488311767578 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 8224 TO DO : save crop sub photo not yet done ! save time : 0.486940860748291 nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 0.5032529830932617 time for calcul the mask position with numpy : 0.01460886001586914 nb_pixel_total : 1064281 time to create 1 rle with new method : 0.03150629997253418 time for calcul the mask position with numpy : 0.0067310333251953125 nb_pixel_total : 4958 time to create 1 rle with old method : 0.0052337646484375 time for calcul the mask position with numpy : 0.006343841552734375 nb_pixel_total : 74956 time to create 1 rle with old method : 0.0828092098236084 time for calcul the mask position with numpy : 0.005928993225097656 nb_pixel_total : 15372 time to create 1 rle with old method : 0.017688274383544922 time for calcul the mask position with numpy : 0.011497974395751953 nb_pixel_total : 753566 time to create 1 rle with new method : 0.09274792671203613 time for calcul the mask position with numpy : 0.0067980289459228516 nb_pixel_total : 110412 time to create 1 rle with old method : 0.12843680381774902 time for calcul the mask position with numpy : 0.006159543991088867 nb_pixel_total : 7584 time to create 1 rle with old method : 0.009215593338012695 time for calcul the mask position with numpy : 0.006481170654296875 nb_pixel_total : 42471 time to create 1 rle with old method : 0.04725337028503418 create new chi : 0.4869711399078369 time to delete rle : 0.0006115436553955078 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 7052 TO DO : save crop sub photo not yet done ! save time : 0.42886829376220703 nb_obj : 13 nb_hashtags : 6 time to prepare the origin masks : 2.087878465652466 time for calcul the mask position with numpy : 0.03747224807739258 nb_pixel_total : 990632 time to create 1 rle with new method : 0.33373260498046875 time for calcul the mask position with numpy : 0.006233692169189453 nb_pixel_total : 1960 time to create 1 rle with old method : 0.0022056102752685547 time for calcul the mask position with numpy : 0.006102561950683594 nb_pixel_total : 35158 time to create 1 rle with old method : 0.038654327392578125 time for calcul the mask position with numpy : 0.00632929801940918 nb_pixel_total : 30359 time to create 1 rle with old method : 0.03532266616821289 time for calcul the mask position with numpy : 0.006243228912353516 nb_pixel_total : 8866 time to create 1 rle with old method : 0.00991201400756836 time for calcul the mask position with numpy : 0.006331443786621094 nb_pixel_total : 48771 time to create 1 rle with old method : 0.053481101989746094 time for calcul the mask position with numpy : 0.006468296051025391 nb_pixel_total : 29826 time to create 1 rle with old method : 0.03305935859680176 time for calcul the mask position with numpy : 0.0061833858489990234 nb_pixel_total : 2337 time to create 1 rle with old method : 0.002645254135131836 time for calcul the mask position with numpy : 0.006012916564941406 nb_pixel_total : 2677 time to create 1 rle with old method : 0.002918243408203125 time for calcul the mask position with numpy : 0.012278079986572266 nb_pixel_total : 821657 time to create 1 rle with new method : 0.18063998222351074 time for calcul the mask position with numpy : 0.006401538848876953 nb_pixel_total : 14540 time to create 1 rle with old method : 0.014911413192749023 time for calcul the mask position with numpy : 0.0062596797943115234 nb_pixel_total : 6993 time to create 1 rle with old method : 0.007747650146484375 time for calcul the mask position with numpy : 0.006306648254394531 nb_pixel_total : 50358 time to create 1 rle with old method : 0.05250835418701172 time for calcul the mask position with numpy : 0.006424665451049805 nb_pixel_total : 29466 time to create 1 rle with old method : 0.03135395050048828 create new chi : 0.9401273727416992 time to delete rle : 0.0007188320159912109 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 7565 TO DO : save crop sub photo not yet done ! save time : 0.44596385955810547 nb_obj : 7 nb_hashtags : 2 time to prepare the origin masks : 0.5218715667724609 time for calcul the mask position with numpy : 0.013191938400268555 nb_pixel_total : 1059934 time to create 1 rle with new method : 0.029810428619384766 time for calcul the mask position with numpy : 0.006005525588989258 nb_pixel_total : 15336 time to create 1 rle with old method : 0.016737699508666992 time for calcul the mask position with numpy : 0.0073473453521728516 nb_pixel_total : 1843 time to create 1 rle with old method : 0.0021653175354003906 time for calcul the mask position with numpy : 0.0059053897857666016 nb_pixel_total : 9855 time to create 1 rle with old method : 0.01112508773803711 time for calcul the mask position with numpy : 0.0058405399322509766 nb_pixel_total : 4222 time to create 1 rle with old method : 0.004853963851928711 time for calcul the mask position with numpy : 0.0061419010162353516 nb_pixel_total : 63875 time to create 1 rle with old method : 0.10833263397216797 time for calcul the mask position with numpy : 0.013377666473388672 nb_pixel_total : 813988 time to create 1 rle with new method : 0.030926227569580078 time for calcul the mask position with numpy : 0.0072784423828125 nb_pixel_total : 104547 time to create 1 rle with old method : 0.12152576446533203 create new chi : 0.391451358795166 time to delete rle : 0.0005385875701904297 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 5584 TO DO : save crop sub photo not yet done ! save time : 0.3385031223297119 map_output_result : {1340106943: (0.0, 'Should be the crop_list due to order', 0), 1340106890: (0.0, 'Should be the crop_list due to order', 0), 1340106888: (0.0, 'Should be the crop_list due to order', 0), 1340106830: (0.0, 'Should be the crop_list due to order', 0), 1340106827: (0.0, 'Should be the crop_list due to order', 0), 1340106725: (0.0, 'Should be the crop_list due to order', 0), 1340106720: (0.0, 'Should be the crop_list due to order', 0), 1340106227: (0.0, 'Should be the crop_list due to order', 0), 1340106225: (0.0, 'Should be the crop_list due to order', 0), 1340105977: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1340106943, 1340106890, 1340106888, 1340106830, 1340106827, 1340106725, 1340106720, 1340106227, 1340106225, 1340105977] Looping around the photos to save general results len do output : 10 /1340106943.Didn't retrieve data . /1340106890.Didn't retrieve data . /1340106888.Didn't retrieve data . /1340106830.Didn't retrieve data . /1340106827.Didn't retrieve data . /1340106725.Didn't retrieve data . /1340106720.Didn't retrieve data . /1340106227.Didn't retrieve data . /1340106225.Didn't retrieve data . /1340105977.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106943', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106890', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106888', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106830', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106827', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106725', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106720', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106227', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106225', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340105977', None, None, None, None, None, '2623036') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01193380355834961 save_final save missing photos in datou_result : time spend for datou_step_exec : 21.680828332901 time spend to save output : 0.012377500534057617 total time spend for step 5 : 21.69320583343506 step6:crop_condition Fri Feb 28 10:09:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 86 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740733782_95702 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8123657703399658 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740733783_95702 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.0496008396148682 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740733785_95702 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6751446723937988 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740733786_95702 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.083806037902832 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740733789_95702 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6209564208984375 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340106943, 1340106890, 1340106888, 1340106830, 1340106827, 1340106725, 1340106720, 1340106227, 1340106225, 1340105977] Looping around the photos to save general results len do output : 13 /1340111343Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340111344Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340111345Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340111346Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340111347Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340111349Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340111350Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340111354Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340111355Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340111356Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340111357Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340111358Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340111360Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106943', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106890', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106888', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106830', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106827', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106725', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106720', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106227', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106225', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340105977', None, None, None, None, None, '2623036') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 49 time used for this insertion : 0.02296757698059082 save_final save missing photos in datou_result : time spend for datou_step_exec : 9.021097421646118 time spend to save output : 0.023479938507080078 total time spend for step 6 : 9.044577360153198 step7:ventilate_hashtags_in_portfolio Fri Feb 28 10:09:50 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20920571 get user id for portfolio 20920571 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20920571 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','papier','Carton_gris','flou','mal_croppe','metal','Carton_brun','environnement','Teint_Dans_La_Masse','autre_refus','plastique','cartonnette')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20920571 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','papier','Carton_gris','flou','mal_croppe','metal','Carton_brun','environnement','Teint_Dans_La_Masse','autre_refus','plastique','cartonnette')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20920674,20920675,20920676,20920677,20920678,20920679,20920680,20920681,20920682,20920683,20920684,20920685?tags=environnement,kraft,cartonnette,plastique,Teint_Dans_La_Masse,flou,metal,papier,mal_croppe,autre_refus,Carton_gris,Carton_brun Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1340106943, 1340106890, 1340106888, 1340106830, 1340106827, 1340106725, 1340106720, 1340106227, 1340106225, 1340105977] Looping around the photos to save general results len do output : 1 /20920571. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106943', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106890', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106888', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106830', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106827', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106725', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106720', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106227', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106225', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340105977', None, None, None, None, None, '2623036') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014371633529663086 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.4365668296813965 time spend to save output : 0.01474142074584961 total time spend for step 7 : 1.451308250427246 step8:final Fri Feb 28 10:09:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1340106943: ('0.0470032056645736',), 1340106890: ('0.0470032056645736',), 1340106888: ('0.0470032056645736',), 1340106830: ('0.0470032056645736',), 1340106827: ('0.0470032056645736',), 1340106725: ('0.0470032056645736',), 1340106720: ('0.0470032056645736',), 1340106227: ('0.0470032056645736',), 1340106225: ('0.0470032056645736',), 1340105977: ('0.0470032056645736',)} new output for save of step final : {1340106943: ('0.0470032056645736',), 1340106890: ('0.0470032056645736',), 1340106888: ('0.0470032056645736',), 1340106830: ('0.0470032056645736',), 1340106827: ('0.0470032056645736',), 1340106725: ('0.0470032056645736',), 1340106720: ('0.0470032056645736',), 1340106227: ('0.0470032056645736',), 1340106225: ('0.0470032056645736',), 1340105977: ('0.0470032056645736',)} [1340106943, 1340106890, 1340106888, 1340106830, 1340106827, 1340106725, 1340106720, 1340106227, 1340106225, 1340105977] Looping around the photos to save general results len do output : 10 /1340106943.Didn't retrieve data . /1340106890.Didn't retrieve data . /1340106888.Didn't retrieve data . /1340106830.Didn't retrieve data . /1340106827.Didn't retrieve data . /1340106725.Didn't retrieve data . /1340106720.Didn't retrieve data . /1340106227.Didn't retrieve data . /1340106225.Didn't retrieve data . /1340105977.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106943', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106890', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106888', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106830', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106827', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106725', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106720', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106227', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106225', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340105977', None, None, None, None, None, '2623036') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.012595891952514648 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11652135848999023 time spend to save output : 0.013086318969726562 total time spend for step 8 : 0.1296076774597168 step9:velours_tree Fri Feb 28 10:09:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.030505895614624023 time spend to save output : 3.933906555175781e-05 total time spend for step 9 : 0.03054523468017578 step10:send_mail_cod Fri Feb 28 10:09:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20920571_28-02-2025_10_09_51.pdf 20920675 change filename to text .imagette209206751740733791 20920676 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209206761740733791 20920677 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209206771740733792 20920679 imagette209206791740733792 20920680 imagette209206801740733792 20920682 imagette209206821740733792 20920683 change filename to text .imagette209206831740733792 20920678 change filename to text .change filename to text .change filename to text .imagette209206781740733792 20920684 imagette209206841740733792 20920685 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209206851740733793 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20920571 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20920674,20920675,20920676,20920677,20920678,20920679,20920680,20920681,20920682,20920683,20920684,20920685?tags=environnement,kraft,cartonnette,plastique,Teint_Dans_La_Masse,flou,metal,papier,mal_croppe,autre_refus,Carton_gris,Carton_brun your option no_mail is active, we will not send the real mail to your client args[1340106943] : ('0.0470032056645736',) no score found for photo 1340106943 We are sending mail with results at report@fotonower.com args[1340106890] : ('0.0470032056645736',) no score found for photo 1340106890 We are sending mail with results at report@fotonower.com args[1340106888] : ('0.0470032056645736',) no score found for photo 1340106888 We are sending mail with results at report@fotonower.com args[1340106830] : ('0.0470032056645736',) no score found for photo 1340106830 We are sending mail with results at report@fotonower.com args[1340106827] : ('0.0470032056645736',) no score found for photo 1340106827 We are sending mail with results at report@fotonower.com args[1340106725] : ('0.0470032056645736',) no score found for photo 1340106725 We are sending mail with results at report@fotonower.com args[1340106720] : ('0.0470032056645736',) no score found for photo 1340106720 We are sending mail with results at report@fotonower.com args[1340106227] : ('0.0470032056645736',) no score found for photo 1340106227 We are sending mail with results at report@fotonower.com args[1340106225] : ('0.0470032056645736',) no score found for photo 1340106225 We are sending mail with results at report@fotonower.com args[1340105977] : ('0.0470032056645736',) no score found for photo 1340105977 We are sending mail with results at report@fotonower.com refus_total : 0.0470032056645736 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20920571 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1340104830,1340105101,1340106227,1340106946,1340106943,1340106890,1340106888,1340106830,1340106827,1340106725,1340106720,1340104779,1340106225,1340105977,1340105724,1340104827,1340104783) Found this number of photos: 17 begin to download photo : 1340104830 begin to download photo : 1340106943 begin to download photo : 1340106827 begin to download photo : 1340106225 begin to download photo : 1340104783 download finish for photo 1340106943 begin to download photo : 1340106890 download finish for photo 1340104783 download finish for photo 1340106827 begin to download photo : 1340106725 download finish for photo 1340104830 begin to download photo : 1340105101 download finish for photo 1340106225 begin to download photo : 1340105977 download finish for photo 1340106725 begin to download photo : 1340106720 download finish for photo 1340106890 begin to download photo : 1340106888 download finish for photo 1340105977 begin to download photo : 1340105724 download finish for photo 1340105101 begin to download photo : 1340106227 download finish for photo 1340106888 begin to download photo : 1340106830 download finish for photo 1340105724 begin to download photo : 1340104827 download finish for photo 1340106227 begin to download photo : 1340106946 download finish for photo 1340106720 begin to download photo : 1340104779 download finish for photo 1340106830 download finish for photo 1340106946 download finish for photo 1340104779 download finish for photo 1340104827 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20920571_28-02-2025_10_09_51.pdf results_Auto_P20920571_28-02-2025_10_09_51.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20920571_28-02-2025_10_09_51.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20920571','results_Auto_P20920571_28-02-2025_10_09_51.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20920571_28-02-2025_10_09_51.pdf','pdf','','0.26','0.0470032056645736') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1340106943, 1340106890, 1340106888, 1340106830, 1340106827, 1340106725, 1340106720, 1340106227, 1340106225, 1340105977] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106943', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106890', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106888', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106830', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106827', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106725', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106720', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106227', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106225', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340105977', None, None, None, None, None, '2623036') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.013675928115844727 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.3217270374298096 time spend to save output : 0.01392054557800293 total time spend for step 10 : 3.3356475830078125 step11:split_time_score Fri Feb 28 10:09:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('09', 26),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 28022025 20920571 Nombre de photos uploadées : 26 / 23040 (0%) 28022025 20920571 Nombre de photos taguées (types de déchets): 0 / 26 (0%) 28022025 20920571 Nombre de photos taguées (volume) : 0 / 26 (0%) elapsed_time : load_data_split_time_score 1.9073486328125e-06 elapsed_time : order_list_meta_photo_and_scores 5.7220458984375e-06 ?????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0014531612396240234 elapsed_time : insert_dashboard_record_day_entry 0.02252960205078125 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.05088624831321179 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20914260_28-02-2025_07_41_57.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20914260 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20914260 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20915778 order by id desc limit 1 Qualite : 0.04017470717428347 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20915779_28-02-2025_08_15_05.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20915779 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20915779 AND mptpi.`type`=3726 To do Qualite : 0.04803433581558472 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20915780_28-02-2025_08_12_03.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20915780 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20915780 AND mptpi.`type`=3726 To do Qualite : 0.06013827796971072 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20918548_28-02-2025_09_29_35.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20918548 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20918548 AND mptpi.`type`=3726 To do Qualite : 0.04300211938089632 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20918549_28-02-2025_09_23_45.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20918549 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20918549 AND mptpi.`type`=3726 To do Qualite : 0.04345018324882482 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20918550_28-02-2025_09_15_51.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20918550 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20918550 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20918551 order by id desc limit 1 Qualite : 0.055121887298477674 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20920571_28-02-2025_10_09_51.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20920571 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20920571 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'28022025': {'nb_upload': 26, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1340106943, 1340106890, 1340106888, 1340106830, 1340106827, 1340106725, 1340106720, 1340106227, 1340106225, 1340105977] Looping around the photos to save general results len do output : 1 /20920571Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106943', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106890', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106888', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106830', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106827', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106725', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106720', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106227', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340106225', None, None, None, None, None, '2623036') ('3459', None, None, None, None, None, None, None, '2623036') ('3459', None, '1340105977', None, None, None, None, None, '2623036') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013036966323852539 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.217961549758911 time spend to save output : 0.0132293701171875 total time spend for step 11 : 2.2311909198760986 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 45.89user 32.71system 1:32.33elapsed 85%CPU (0avgtext+0avgdata 2823672maxresident)k 48176inputs+32224outputs (261major+2190137minor)pagefaults 0swaps