python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2145489 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 40>10 please execute split_portfolio.py -i 20596788 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2592316'] with mtr_portfolio_ids : ['20596788'] and first list_photo_ids : [] new path : /proc/2145489/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.596893548965454 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Mon Feb 17 07:32:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10998 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-17 07:32:31.724830: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-17 07:32:31.751108: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-17 07:32:31.753157: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8284000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-17 07:32:31.753208: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-17 07:32:31.756846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-17 07:32:32.072664: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2ece0ec0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-17 07:32:32.072713: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-17 07:32:32.074115: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-17 07:32:32.078193: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-17 07:32:32.082347: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-17 07:32:32.085026: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-17 07:32:32.086271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-17 07:32:32.090140: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-17 07:32:32.091656: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-17 07:32:32.098283: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-17 07:32:32.100027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-17 07:32:32.100088: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-17 07:32:32.100862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-17 07:32:32.100877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-17 07:32:32.100885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-17 07:32:32.102276: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10193 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-17 07:32:32.364520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-17 07:32:32.364633: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-17 07:32:32.364654: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-17 07:32:32.364671: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-17 07:32:32.364688: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-17 07:32:32.364704: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-17 07:32:32.364720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-17 07:32:32.364736: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-17 07:32:32.366060: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-17 07:32:32.367393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-17 07:32:32.367490: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-17 07:32:32.367509: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-17 07:32:32.367525: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-17 07:32:32.367541: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-17 07:32:32.367557: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-17 07:32:32.367573: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-17 07:32:32.367589: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-17 07:32:32.368946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-17 07:32:32.368992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-17 07:32:32.369001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-17 07:32:32.369009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-17 07:32:32.370430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10193 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-17 07:32:42.975101: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-17 07:32:43.160463: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 138.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 2.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 138.60000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 7.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 138.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 2.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 138.60000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 138.85000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 141.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 144.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 141.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 138.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 1 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 139.85000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 11 Detection mask done ! Trying to reset tf kernel 2146020 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5706 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10998 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0132904052734375 nb_pixel_total : 532591 time to create 1 rle with new method : 0.026019573211669922 length of segment : 1939 time for calcul the mask position with numpy : 0.0023145675659179688 nb_pixel_total : 17903 time to create 1 rle with old method : 0.0231783390045166 length of segment : 402 time for calcul the mask position with numpy : 0.0002644062042236328 nb_pixel_total : 4296 time to create 1 rle with old method : 0.005261421203613281 length of segment : 73 time for calcul the mask position with numpy : 0.03558039665222168 nb_pixel_total : 1829455 time to create 1 rle with new method : 0.07761144638061523 length of segment : 2134 time for calcul the mask position with numpy : 0.001399993896484375 nb_pixel_total : 69179 time to create 1 rle with old method : 0.07392454147338867 length of segment : 264 time for calcul the mask position with numpy : 0.001153707504272461 nb_pixel_total : 80865 time to create 1 rle with old method : 0.08818769454956055 length of segment : 267 time for calcul the mask position with numpy : 0.0009539127349853516 nb_pixel_total : 72915 time to create 1 rle with old method : 0.07876944541931152 length of segment : 259 time for calcul the mask position with numpy : 0.0006265640258789062 nb_pixel_total : 29976 time to create 1 rle with old method : 0.032010793685913086 length of segment : 275 time for calcul the mask position with numpy : 0.000278472900390625 nb_pixel_total : 14529 time to create 1 rle with old method : 0.015683650970458984 length of segment : 178 time for calcul the mask position with numpy : 0.023921966552734375 nb_pixel_total : 1640990 time to create 1 rle with new method : 0.0550384521484375 length of segment : 2626 time for calcul the mask position with numpy : 0.0010123252868652344 nb_pixel_total : 74186 time to create 1 rle with old method : 0.07852625846862793 length of segment : 279 time for calcul the mask position with numpy : 0.007737159729003906 nb_pixel_total : 430472 time to create 1 rle with new method : 0.018216609954833984 length of segment : 985 time for calcul the mask position with numpy : 0.0012180805206298828 nb_pixel_total : 91427 time to create 1 rle with old method : 0.09902501106262207 length of segment : 699 time for calcul the mask position with numpy : 0.00035881996154785156 nb_pixel_total : 16640 time to create 1 rle with old method : 0.019065380096435547 length of segment : 117 time for calcul the mask position with numpy : 0.0050199031829833984 nb_pixel_total : 139070 time to create 1 rle with old method : 0.14902687072753906 length of segment : 1296 time for calcul the mask position with numpy : 0.0019176006317138672 nb_pixel_total : 159967 time to create 1 rle with new method : 0.004662990570068359 length of segment : 541 time for calcul the mask position with numpy : 0.02002239227294922 nb_pixel_total : 1200148 time to create 1 rle with new method : 0.053904056549072266 length of segment : 2460 time for calcul the mask position with numpy : 0.0005462169647216797 nb_pixel_total : 35347 time to create 1 rle with old method : 0.03748893737792969 length of segment : 250 time for calcul the mask position with numpy : 0.011089563369750977 nb_pixel_total : 699435 time to create 1 rle with new method : 0.024092912673950195 length of segment : 1287 time for calcul the mask position with numpy : 0.008026361465454102 nb_pixel_total : 533430 time to create 1 rle with new method : 0.018440961837768555 length of segment : 1866 time for calcul the mask position with numpy : 0.0010874271392822266 nb_pixel_total : 95969 time to create 1 rle with old method : 0.10261154174804688 length of segment : 447 time for calcul the mask position with numpy : 0.0038084983825683594 nb_pixel_total : 307775 time to create 1 rle with new method : 0.010593891143798828 length of segment : 1854 time for calcul the mask position with numpy : 0.0017135143280029297 nb_pixel_total : 156575 time to create 1 rle with new method : 0.003396272659301758 length of segment : 461 time for calcul the mask position with numpy : 0.0002372264862060547 nb_pixel_total : 12641 time to create 1 rle with old method : 0.01380610466003418 length of segment : 154 time for calcul the mask position with numpy : 0.009511947631835938 nb_pixel_total : 444729 time to create 1 rle with new method : 0.029937267303466797 length of segment : 2473 time for calcul the mask position with numpy : 0.0028209686279296875 nb_pixel_total : 178844 time to create 1 rle with new method : 0.005995035171508789 length of segment : 619 time for calcul the mask position with numpy : 0.004642963409423828 nb_pixel_total : 192116 time to create 1 rle with new method : 0.015038728713989258 length of segment : 1099 time for calcul the mask position with numpy : 0.00045490264892578125 nb_pixel_total : 12121 time to create 1 rle with old method : 0.018038511276245117 length of segment : 125 time for calcul the mask position with numpy : 0.0005919933319091797 nb_pixel_total : 22890 time to create 1 rle with old method : 0.025643348693847656 length of segment : 147 time for calcul the mask position with numpy : 0.0004930496215820312 nb_pixel_total : 15581 time to create 1 rle with old method : 0.017737865447998047 length of segment : 180 time for calcul the mask position with numpy : 0.0003268718719482422 nb_pixel_total : 15482 time to create 1 rle with old method : 0.017588138580322266 length of segment : 93 time for calcul the mask position with numpy : 0.0032029151916503906 nb_pixel_total : 168239 time to create 1 rle with new method : 0.005098819732666016 length of segment : 835 time for calcul the mask position with numpy : 0.0027608871459960938 nb_pixel_total : 172726 time to create 1 rle with new method : 0.004972934722900391 length of segment : 418 time for calcul the mask position with numpy : 0.002797365188598633 nb_pixel_total : 125985 time to create 1 rle with old method : 0.1329333782196045 length of segment : 1082 time for calcul the mask position with numpy : 0.0015177726745605469 nb_pixel_total : 31039 time to create 1 rle with old method : 0.0367741584777832 length of segment : 361 time for calcul the mask position with numpy : 0.0032122135162353516 nb_pixel_total : 163367 time to create 1 rle with new method : 0.004876613616943359 length of segment : 1036 time for calcul the mask position with numpy : 0.005129337310791016 nb_pixel_total : 198747 time to create 1 rle with new method : 0.010153770446777344 length of segment : 486 time for calcul the mask position with numpy : 0.000392913818359375 nb_pixel_total : 8680 time to create 1 rle with old method : 0.010271787643432617 length of segment : 148 time for calcul the mask position with numpy : 0.013718128204345703 nb_pixel_total : 863395 time to create 1 rle with new method : 0.02054905891418457 length of segment : 1959 time for calcul the mask position with numpy : 0.0014684200286865234 nb_pixel_total : 69621 time to create 1 rle with old method : 0.07565760612487793 length of segment : 380 time for calcul the mask position with numpy : 0.0005311965942382812 nb_pixel_total : 25474 time to create 1 rle with old method : 0.02874135971069336 length of segment : 312 time for calcul the mask position with numpy : 0.0014009475708007812 nb_pixel_total : 89645 time to create 1 rle with old method : 0.09746408462524414 length of segment : 307 time for calcul the mask position with numpy : 0.003807544708251953 nb_pixel_total : 196285 time to create 1 rle with new method : 0.005606651306152344 length of segment : 1096 time for calcul the mask position with numpy : 0.00829315185546875 nb_pixel_total : 475637 time to create 1 rle with new method : 0.014575719833374023 length of segment : 788 time for calcul the mask position with numpy : 0.0043141841888427734 nb_pixel_total : 103260 time to create 1 rle with old method : 0.1145944595336914 length of segment : 461 time for calcul the mask position with numpy : 0.008874893188476562 nb_pixel_total : 500125 time to create 1 rle with new method : 0.014940261840820312 length of segment : 789 time for calcul the mask position with numpy : 0.00370025634765625 nb_pixel_total : 177030 time to create 1 rle with new method : 0.007402181625366211 length of segment : 441 time for calcul the mask position with numpy : 0.0048902034759521484 nb_pixel_total : 364651 time to create 1 rle with new method : 0.013257503509521484 length of segment : 800 time for calcul the mask position with numpy : 0.0037887096405029297 nb_pixel_total : 267930 time to create 1 rle with new method : 0.006942272186279297 length of segment : 566 time for calcul the mask position with numpy : 0.0007236003875732422 nb_pixel_total : 36596 time to create 1 rle with old method : 0.04079031944274902 length of segment : 303 time for calcul the mask position with numpy : 0.0047495365142822266 nb_pixel_total : 168081 time to create 1 rle with new method : 0.008185148239135742 length of segment : 690 time for calcul the mask position with numpy : 0.0007026195526123047 nb_pixel_total : 19477 time to create 1 rle with old method : 0.022907257080078125 length of segment : 174 time for calcul the mask position with numpy : 0.002004384994506836 nb_pixel_total : 36978 time to create 1 rle with old method : 0.04158282279968262 length of segment : 384 time for calcul the mask position with numpy : 0.003308534622192383 nb_pixel_total : 153328 time to create 1 rle with new method : 0.0051479339599609375 length of segment : 706 time for calcul the mask position with numpy : 0.003252744674682617 nb_pixel_total : 176192 time to create 1 rle with new method : 0.008702754974365234 length of segment : 715 time for calcul the mask position with numpy : 0.0026140213012695312 nb_pixel_total : 103785 time to create 1 rle with old method : 0.11818122863769531 length of segment : 622 time for calcul the mask position with numpy : 0.001260995864868164 nb_pixel_total : 55246 time to create 1 rle with old method : 0.0602724552154541 length of segment : 230 time for calcul the mask position with numpy : 0.001708984375 nb_pixel_total : 59634 time to create 1 rle with old method : 0.06571173667907715 length of segment : 419 time for calcul the mask position with numpy : 0.0006594657897949219 nb_pixel_total : 23280 time to create 1 rle with old method : 0.026170969009399414 length of segment : 279 time spent for convertir_results : 8.809476137161255 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 59 chid ids of type : 3663 Number RLEs to save : 42636 save missing photos in datou_result : time spend for datou_step_exec : 49.05905604362488 time spend to save output : 2.358551502227783 total time spend for step 1 : 51.41760754585266 step2:crop_condition Mon Feb 17 07:33:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 59 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1338117067, 1338117060, 1338117036, 1338117028, 1338117016, 1338117009, 1338116988, 1338116984, 1338116970, 1338116959] Looping around the photos to save general results len do output : 34 /-3677712645Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712650Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712657Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712666Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712662Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712669Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712679Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712681Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712680Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712685Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712687Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712640Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712652Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712676Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712688Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712675Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712636Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712639Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712647Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712653Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712667Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712671Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712674Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712683Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712690Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712689Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712691Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712637Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712686Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712641Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712655Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712658Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712678Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677712692Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117067', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117060', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117036', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117028', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117016', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117009', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116988', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116984', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116970', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116959', None, None, None, None, None, '2592316') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 112 time used for this insertion : 0.02464151382446289 save_final save missing photos in datou_result : time spend for datou_step_exec : 27.442764282226562 time spend to save output : 0.02567458152770996 total time spend for step 2 : 27.468438863754272 step3:thcl Mon Feb 17 07:33:47 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.0077397823333740234 time to convert the images to numpy array : 5.7220458984375e-06 time to import caffe and check if the image exist : 0.016234874725341797 time to convert the images to numpy array : 0.03820395469665527 time to import caffe and check if the image exist : 0.014026641845703125 time to convert the images to numpy array : 0.06812906265258789 time to import caffe and check if the image exist : 0.015070676803588867 time to convert the images to numpy array : 0.10037589073181152 time to import caffe and check if the image exist : 0.009750127792358398 time to convert the images to numpy array : 0.10691022872924805 time to import caffe and check if the image exist : 0.009440183639526367 time to convert the images to numpy array : 0.10810065269470215 time to import caffe and check if the image exist : 0.013486862182617188 time to convert the images to numpy array : 0.12342977523803711 time to import caffe and check if the image exist : 0.012575626373291016 time to convert the images to numpy array : 0.1253516674041748 time to import caffe and check if the image exist : 0.011530637741088867 time to convert the images to numpy array : 0.12810397148132324 time to import caffe and check if the image exist : 0.013324499130249023 time to convert the images to numpy array : 0.17940545082092285 total time to convert the images to numpy array : 0.6027083396911621 list photo_ids error: [] list photo_ids correct : [-3677712686, -3677712641, -3677712655, -3677712658, -3677712652, -3677712676, -3677712688, -3677712675, -3677712690, -3677712689, -3677712691, -3677712637, -3677712645, -3677712650, -3677712657, -3677712666, -3677712662, -3677712669, -3677712679, -3677712681, -3677712678, -3677712692, -3677712667, -3677712671, -3677712674, -3677712683, -3677712680, -3677712685, -3677712687, -3677712640, -3677712636, -3677712639, -3677712647, -3677712653] number of photos to traite : 34 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10998 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10776 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.6190662384033203 time used to do the prediction : 0.14550566673278809 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.5979924201965332 time used to do the prediction : 0.08571195602416992 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.0251998901367188e-05 save missing photos in datou_result : time spend for datou_step_exec : 9.036191463470459 time spend to save output : 0.00030732154846191406 total time spend for step 3 : 9.036498785018921 step4:merge_mask_thcl_custom Mon Feb 17 07:33:56 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 59 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1338117067, 1338117060, 1338117036, 1338117028, 1338117016, 1338117009, 1338116988, 1338116984, 1338116970, 1338116959] Looping around the photos to save general results len do output : 10 /1338117067Didn't retrieve data .Didn't retrieve data . /1338117060Didn't retrieve data .Didn't retrieve data . /1338117036Didn't retrieve data .Didn't retrieve data . /1338117028Didn't retrieve data .Didn't retrieve data . /1338117016Didn't retrieve data .Didn't retrieve data . /1338117009Didn't retrieve data .Didn't retrieve data . /1338116988Didn't retrieve data .Didn't retrieve data . /1338116984Didn't retrieve data .Didn't retrieve data . /1338116970Didn't retrieve data .Didn't retrieve data . /1338116959Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117067', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117060', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117036', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117028', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117016', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117009', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116988', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116984', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116970', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116959', None, None, None, None, None, '2592316') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014673709869384766 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.05935192108154297 time spend to save output : 0.015202760696411133 total time spend for step 4 : 0.0745546817779541 step5:rle_unique_nms_with_priority Mon Feb 17 07:33:56 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 5.150936841964722 time for calcul the mask position with numpy : 0.4125239849090576 nb_pixel_total : 5832209 time to create 1 rle with new method : 0.8129520416259766 time for calcul the mask position with numpy : 0.024950504302978516 nb_pixel_total : 925 time to create 1 rle with old method : 0.0013303756713867188 time for calcul the mask position with numpy : 0.024559736251831055 nb_pixel_total : 12117 time to create 1 rle with old method : 0.013268232345581055 time for calcul the mask position with numpy : 0.024723291397094727 nb_pixel_total : 68916 time to create 1 rle with old method : 0.0720663070678711 time for calcul the mask position with numpy : 0.03725171089172363 nb_pixel_total : 1826314 time to create 1 rle with new method : 0.5925459861755371 time for calcul the mask position with numpy : 0.02690601348876953 nb_pixel_total : 4269 time to create 1 rle with old method : 0.0063724517822265625 time for calcul the mask position with numpy : 0.02902960777282715 nb_pixel_total : 531840 time to create 1 rle with new method : 0.5705220699310303 time for calcul the mask position with numpy : 0.025104761123657227 nb_pixel_total : 17810 time to create 1 rle with old method : 0.019108057022094727 create new chi : 2.784731864929199 time to delete rle : 0.009852409362792969 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 12462 TO DO : save crop sub photo not yet done ! save time : 0.688326358795166 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 3.7353997230529785 time for calcul the mask position with numpy : 0.46944093704223633 nb_pixel_total : 6379568 time to create 1 rle with new method : 0.7842085361480713 time for calcul the mask position with numpy : 0.04489254951477051 nb_pixel_total : 73964 time to create 1 rle with old method : 0.08231830596923828 time for calcul the mask position with numpy : 0.04280233383178711 nb_pixel_total : 16551 time to create 1 rle with old method : 0.017409563064575195 time for calcul the mask position with numpy : 0.041638851165771484 nb_pixel_total : 94997 time to create 1 rle with old method : 0.10286879539489746 time for calcul the mask position with numpy : 0.056420326232910156 nb_pixel_total : 1637944 time to create 1 rle with new method : 0.8614413738250732 time for calcul the mask position with numpy : 0.042079925537109375 nb_pixel_total : 91376 time to create 1 rle with old method : 0.09493851661682129 create new chi : 2.7219910621643066 time to delete rle : 0.0008721351623535156 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 10206 TO DO : save crop sub photo not yet done ! save time : 0.5652894973754883 nb_obj : 6 nb_hashtags : 4 time to prepare the origin masks : 3.7211124897003174 time for calcul the mask position with numpy : 0.3148670196533203 nb_pixel_total : 6012171 time to create 1 rle with new method : 0.6059715747833252 time for calcul the mask position with numpy : 0.026108980178833008 nb_pixel_total : 258933 time to create 1 rle with new method : 0.6277062892913818 time for calcul the mask position with numpy : 0.024701356887817383 nb_pixel_total : 35184 time to create 1 rle with old method : 0.03661751747131348 time for calcul the mask position with numpy : 0.029036521911621094 nb_pixel_total : 532711 time to create 1 rle with new method : 0.42627668380737305 time for calcul the mask position with numpy : 0.0274813175201416 nb_pixel_total : 95740 time to create 1 rle with old method : 0.09647107124328613 time for calcul the mask position with numpy : 0.03958559036254883 nb_pixel_total : 159814 time to create 1 rle with new method : 0.5281286239624023 time for calcul the mask position with numpy : 0.05072021484375 nb_pixel_total : 1199847 time to create 1 rle with new method : 0.45886802673339844 create new chi : 3.4391398429870605 time to delete rle : 0.001445770263671875 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 15063 TO DO : save crop sub photo not yet done ! save time : 0.8274767398834229 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 3.4177796840667725 time for calcul the mask position with numpy : 0.362027645111084 nb_pixel_total : 7252531 time to create 1 rle with new method : 0.9704196453094482 time for calcul the mask position with numpy : 0.029296398162841797 nb_pixel_total : 156435 time to create 1 rle with new method : 0.7728590965270996 time for calcul the mask position with numpy : 0.02823948860168457 nb_pixel_total : 121010 time to create 1 rle with old method : 0.1304931640625 time for calcul the mask position with numpy : 0.029470443725585938 nb_pixel_total : 444041 time to create 1 rle with new method : 0.589148759841919 time for calcul the mask position with numpy : 0.0268707275390625 nb_pixel_total : 12594 time to create 1 rle with old method : 0.01375889778137207 time for calcul the mask position with numpy : 0.030058622360229492 nb_pixel_total : 307789 time to create 1 rle with new method : 0.5582845211029053 create new chi : 3.745436429977417 time to delete rle : 0.0017049312591552734 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 12941 TO DO : save crop sub photo not yet done ! save time : 0.7323801517486572 nb_obj : 6 nb_hashtags : 3 time to prepare the origin masks : 3.3716065883636475 time for calcul the mask position with numpy : 0.4500541687011719 nb_pixel_total : 7880527 time to create 1 rle with new method : 0.6208541393280029 time for calcul the mask position with numpy : 0.024106979370117188 nb_pixel_total : 12040 time to create 1 rle with old method : 0.012495994567871094 time for calcul the mask position with numpy : 0.024669170379638672 nb_pixel_total : 172045 time to create 1 rle with new method : 0.5084335803985596 time for calcul the mask position with numpy : 0.0242307186126709 nb_pixel_total : 15422 time to create 1 rle with old method : 0.01554250717163086 time for calcul the mask position with numpy : 0.025158405303955078 nb_pixel_total : 176013 time to create 1 rle with new method : 0.6979587078094482 time for calcul the mask position with numpy : 0.026656389236450195 nb_pixel_total : 15557 time to create 1 rle with old method : 0.016983509063720703 time for calcul the mask position with numpy : 0.02478623390197754 nb_pixel_total : 22796 time to create 1 rle with old method : 0.02407383918762207 create new chi : 2.60745906829834 time to delete rle : 0.0005240440368652344 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 5378 TO DO : save crop sub photo not yet done ! save time : 0.3212928771972656 nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 2.462963581085205 time for calcul the mask position with numpy : 0.3484365940093994 nb_pixel_total : 7776269 time to create 1 rle with new method : 0.9290957450866699 time for calcul the mask position with numpy : 0.034444332122802734 nb_pixel_total : 163154 time to create 1 rle with new method : 0.5965900421142578 time for calcul the mask position with numpy : 0.027403831481933594 nb_pixel_total : 198105 time to create 1 rle with new method : 0.5904481410980225 time for calcul the mask position with numpy : 0.027040481567382812 nb_pixel_total : 125907 time to create 1 rle with old method : 0.1592414379119873 time for calcul the mask position with numpy : 0.03045654296875 nb_pixel_total : 30965 time to create 1 rle with old method : 0.032990455627441406 create new chi : 2.8792154788970947 time to delete rle : 0.0008845329284667969 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 8098 TO DO : save crop sub photo not yet done ! save time : 0.4671187400817871 nb_obj : 4 nb_hashtags : 4 time to prepare the origin masks : 3.709444284439087 time for calcul the mask position with numpy : 0.6893560886383057 nb_pixel_total : 7247341 time to create 1 rle with new method : 1.3747541904449463 time for calcul the mask position with numpy : 0.025663375854492188 nb_pixel_total : 69417 time to create 1 rle with old method : 0.07385611534118652 time for calcul the mask position with numpy : 0.026390552520751953 nb_pixel_total : 89462 time to create 1 rle with old method : 0.09530520439147949 time for calcul the mask position with numpy : 0.033190250396728516 nb_pixel_total : 862433 time to create 1 rle with new method : 0.5863988399505615 time for calcul the mask position with numpy : 0.024179935455322266 nb_pixel_total : 25747 time to create 1 rle with old method : 0.02760601043701172 create new chi : 3.034421920776367 time to delete rle : 0.0007827281951904297 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 8012 TO DO : save crop sub photo not yet done ! save time : 0.47113704681396484 nb_obj : 6 nb_hashtags : 3 time to prepare the origin masks : 3.918943405151367 time for calcul the mask position with numpy : 0.4734220504760742 nb_pixel_total : 6577479 time to create 1 rle with new method : 1.0731761455535889 time for calcul the mask position with numpy : 0.03739500045776367 nb_pixel_total : 474448 time to create 1 rle with new method : 0.5713155269622803 time for calcul the mask position with numpy : 0.03910565376281738 nb_pixel_total : 267310 time to create 1 rle with new method : 0.5099992752075195 time for calcul the mask position with numpy : 0.04081916809082031 nb_pixel_total : 196118 time to create 1 rle with new method : 0.5580499172210693 time for calcul the mask position with numpy : 0.04357314109802246 nb_pixel_total : 499151 time to create 1 rle with new method : 0.763878583908081 time for calcul the mask position with numpy : 0.04076647758483887 nb_pixel_total : 176815 time to create 1 rle with new method : 0.8323466777801514 time for calcul the mask position with numpy : 0.04120278358459473 nb_pixel_total : 103079 time to create 1 rle with old method : 0.10871100425720215 create new chi : 5.326765537261963 time to delete rle : 0.0012621879577636719 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 10466 TO DO : save crop sub photo not yet done ! save time : 0.5837376117706299 nb_obj : 1 nb_hashtags : 1 time to prepare the origin masks : 1.0536208152770996 time for calcul the mask position with numpy : 0.5303542613983154 nb_pixel_total : 8257866 time to create 1 rle with new method : 0.6096804141998291 time for calcul the mask position with numpy : 0.032588958740234375 nb_pixel_total : 36534 time to create 1 rle with old method : 0.04139542579650879 create new chi : 1.2624642848968506 time to delete rle : 0.000286102294921875 batch 1 Loaded 2 chid ids of type : 3726 Number RLEs to save : 2770 TO DO : save crop sub photo not yet done ! save time : 0.2085285186767578 nb_obj : 8 nb_hashtags : 3 time to prepare the origin masks : 5.72006368637085 time for calcul the mask position with numpy : 0.5450043678283691 nb_pixel_total : 7686833 time to create 1 rle with new method : 1.0747807025909424 time for calcul the mask position with numpy : 0.03971052169799805 nb_pixel_total : 59493 time to create 1 rle with old method : 0.06229281425476074 time for calcul the mask position with numpy : 0.04421591758728027 nb_pixel_total : 106476 time to create 1 rle with old method : 0.11205315589904785 time for calcul the mask position with numpy : 0.043166399002075195 nb_pixel_total : 55132 time to create 1 rle with old method : 0.05777597427368164 time for calcul the mask position with numpy : 0.03924560546875 nb_pixel_total : 36821 time to create 1 rle with old method : 0.04214978218078613 time for calcul the mask position with numpy : 0.031075239181518555 nb_pixel_total : 19400 time to create 1 rle with old method : 0.02673172950744629 time for calcul the mask position with numpy : 0.037673234939575195 nb_pixel_total : 9334 time to create 1 rle with old method : 0.015068292617797852 time for calcul the mask position with numpy : 0.03598594665527344 nb_pixel_total : 167884 time to create 1 rle with new method : 0.6263751983642578 time for calcul the mask position with numpy : 0.028072118759155273 nb_pixel_total : 153027 time to create 1 rle with new method : 0.49329686164855957 create new chi : 3.457066535949707 time to delete rle : 0.0010638236999511719 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 9489 TO DO : save crop sub photo not yet done ! save time : 0.653362512588501 map_output_result : {1338117067: (0.0, 'Should be the crop_list due to order', 0.0), 1338117060: (0.0, 'Should be the crop_list due to order', 0.0), 1338117036: (0.0, 'Should be the crop_list due to order', 0.0), 1338117028: (0.0, 'Should be the crop_list due to order', 0.0), 1338117016: (0.0, 'Should be the crop_list due to order', 0.0), 1338117009: (0.0, 'Should be the crop_list due to order', 0.0), 1338116988: (0.0, 'Should be the crop_list due to order', 0.0), 1338116984: (0.0, 'Should be the crop_list due to order', 0.0), 1338116970: (0.0, 'Should be the crop_list due to order', 0.0), 1338116959: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1338117067, 1338117060, 1338117036, 1338117028, 1338117016, 1338117009, 1338116988, 1338116984, 1338116970, 1338116959] Looping around the photos to save general results len do output : 10 /1338117067.Didn't retrieve data . /1338117060.Didn't retrieve data . /1338117036.Didn't retrieve data . /1338117028.Didn't retrieve data . /1338117016.Didn't retrieve data . /1338117009.Didn't retrieve data . /1338116988.Didn't retrieve data . /1338116984.Didn't retrieve data . /1338116970.Didn't retrieve data . /1338116959.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117067', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117060', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117036', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117028', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117016', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117009', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116988', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116984', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116970', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116959', None, None, None, None, None, '2592316') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01443171501159668 save_final save missing photos in datou_result : time spend for datou_step_exec : 74.14043641090393 time spend to save output : 0.015050888061523438 total time spend for step 5 : 74.15548729896545 step6:crop_condition Mon Feb 17 07:35:11 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 62 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739774117_2145489 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.7861034870147705 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739774121_2145489 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.9926252365112305 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739774124_2145489 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5012598037719727 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1338117067, 1338117060, 1338117036, 1338117028, 1338117016, 1338117009, 1338116988, 1338116984, 1338116970, 1338116959] Looping around the photos to save general results len do output : 9 /1338117650Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338117651Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338117652Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338117654Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338117659Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338117660Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338117661Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338117662Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338117665Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117067', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117060', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117036', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117028', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117016', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117009', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116988', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116984', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116970', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116959', None, None, None, None, None, '2592316') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 37 time used for this insertion : 0.028946876525878906 save_final save missing photos in datou_result : time spend for datou_step_exec : 14.81152057647705 time spend to save output : 0.029644489288330078 total time spend for step 6 : 14.84116506576538 step7:ventilate_hashtags_in_portfolio Mon Feb 17 07:35:25 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20596788 get user id for portfolio 20596788 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20596788 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_gris','kraft','mal_croppe','flou','cartonnette','Carton_brun','autre_refus','plastique','Teint_Dans_La_Masse','environnement','papier','metal')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20596788 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_gris','kraft','mal_croppe','flou','cartonnette','Carton_brun','autre_refus','plastique','Teint_Dans_La_Masse','environnement','papier','metal')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20596788 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_gris','kraft','mal_croppe','flou','cartonnette','Carton_brun','autre_refus','plastique','Teint_Dans_La_Masse','environnement','papier','metal')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20596985,20596986,20596987,20596988,20596989,20596990,20596991,20596992,20596993,20596994,20596995,20596996?tags=Carton_gris,kraft,mal_croppe,flou,cartonnette,Carton_brun,autre_refus,plastique,Teint_Dans_La_Masse,environnement,papier,metal Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1338117067, 1338117060, 1338117036, 1338117028, 1338117016, 1338117009, 1338116988, 1338116984, 1338116970, 1338116959] Looping around the photos to save general results len do output : 1 /20596788. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117067', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117060', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117036', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117028', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117016', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117009', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116988', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116984', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116970', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116959', None, None, None, None, None, '2592316') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014225959777832031 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.6908740997314453 time spend to save output : 0.01454925537109375 total time spend for step 7 : 1.705423355102539 step8:final Mon Feb 17 07:35:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1338117067: ('0.03486663619902253',), 1338117060: ('0.03486663619902253',), 1338117036: ('0.03486663619902253',), 1338117028: ('0.03486663619902253',), 1338117016: ('0.03486663619902253',), 1338117009: ('0.03486663619902253',), 1338116988: ('0.03486663619902253',), 1338116984: ('0.03486663619902253',), 1338116970: ('0.03486663619902253',), 1338116959: ('0.03486663619902253',)} new output for save of step final : {1338117067: ('0.03486663619902253',), 1338117060: ('0.03486663619902253',), 1338117036: ('0.03486663619902253',), 1338117028: ('0.03486663619902253',), 1338117016: ('0.03486663619902253',), 1338117009: ('0.03486663619902253',), 1338116988: ('0.03486663619902253',), 1338116984: ('0.03486663619902253',), 1338116970: ('0.03486663619902253',), 1338116959: ('0.03486663619902253',)} [1338117067, 1338117060, 1338117036, 1338117028, 1338117016, 1338117009, 1338116988, 1338116984, 1338116970, 1338116959] Looping around the photos to save general results len do output : 10 /1338117067.Didn't retrieve data . /1338117060.Didn't retrieve data . /1338117036.Didn't retrieve data . /1338117028.Didn't retrieve data . /1338117016.Didn't retrieve data . /1338117009.Didn't retrieve data . /1338116988.Didn't retrieve data . /1338116984.Didn't retrieve data . /1338116970.Didn't retrieve data . /1338116959.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117067', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117060', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117036', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117028', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117016', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117009', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116988', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116984', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116970', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116959', None, None, None, None, None, '2592316') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.0198214054107666 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.12395429611206055 time spend to save output : 0.02034783363342285 total time spend for step 8 : 0.1443021297454834 step9:velours_tree Mon Feb 17 07:35:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.03379106521606445 time spend to save output : 5.221366882324219e-05 total time spend for step 9 : 0.033843278884887695 step10:send_mail_cod Mon Feb 17 07:35:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20596788_17-02-2025_07_35_27.pdf 20596986 imagette205969861739774127 20596987 imagette205969871739774127 20596988 imagette205969881739774127 20596989 imagette205969891739774127 20596991 change filename to text .change filename to text .change filename to text .change filename to text .imagette205969911739774127 20596992 imagette205969921739774128 20596996 imagette205969961739774128 20596985 imagette205969851739774128 20596990 change filename to text .imagette205969901739774128 20596993 change filename to text .change filename to text .change filename to text .change filename to text .imagette205969931739774128 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20596788 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20596985,20596986,20596987,20596988,20596989,20596990,20596991,20596992,20596993,20596994,20596995,20596996?tags=Carton_gris,kraft,mal_croppe,flou,cartonnette,Carton_brun,autre_refus,plastique,Teint_Dans_La_Masse,environnement,papier,metal your option no_mail is active, we will not send the real mail to your client args[1338117067] : ('0.03486663619902253',) no score found for photo 1338117067 apple ('0.03486663619902253',) We are sending mail with results at report@fotonower.com args[1338117060] : ('0.03486663619902253',) no score found for photo 1338117060 apple ('0.03486663619902253',) We are sending mail with results at report@fotonower.com args[1338117036] : ('0.03486663619902253',) no score found for photo 1338117036 apple ('0.03486663619902253',) We are sending mail with results at report@fotonower.com args[1338117028] : ('0.03486663619902253',) no score found for photo 1338117028 apple ('0.03486663619902253',) We are sending mail with results at report@fotonower.com args[1338117016] : ('0.03486663619902253',) no score found for photo 1338117016 apple ('0.03486663619902253',) We are sending mail with results at report@fotonower.com args[1338117009] : ('0.03486663619902253',) no score found for photo 1338117009 apple ('0.03486663619902253',) We are sending mail with results at report@fotonower.com args[1338116988] : ('0.03486663619902253',) no score found for photo 1338116988 apple ('0.03486663619902253',) We are sending mail with results at report@fotonower.com args[1338116984] : ('0.03486663619902253',) no score found for photo 1338116984 apple ('0.03486663619902253',) We are sending mail with results at report@fotonower.com args[1338116970] : ('0.03486663619902253',) no score found for photo 1338116970 apple ('0.03486663619902253',) We are sending mail with results at report@fotonower.com args[1338116959] : ('0.03486663619902253',) no score found for photo 1338116959 apple ('0.03486663619902253',) We are sending mail with results at report@fotonower.com refus_total : 0.03486663619902253 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20596788 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20596788_17-02-2025_07_35_27.pdf results_Auto_P20596788_17-02-2025_07_35_27.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20596788_17-02-2025_07_35_27.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20596788','results_Auto_P20596788_17-02-2025_07_35_27.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20596788_17-02-2025_07_35_27.pdf','pdf','','0.43','0.03486663619902253') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1338117067, 1338117060, 1338117036, 1338117028, 1338117016, 1338117009, 1338116988, 1338116984, 1338116970, 1338116959] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117067', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117060', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117036', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117028', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117016', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117009', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116988', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116984', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116970', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116959', None, None, None, None, None, '2592316') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.013545751571655273 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.1492090225219727 time spend to save output : 0.013876676559448242 total time spend for step 10 : 3.163085699081421 step11:split_time_score Mon Feb 17 07:35:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('07', 40),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 17022025 20596788 Nombre de photos uploadées : 40 / 23040 (0%) 17022025 20596788 Nombre de photos taguées (types de déchets): 0 / 40 (0%) 17022025 20596788 Nombre de photos taguées (volume) : 0 / 40 (0%) elapsed_time : load_data_split_time_score 1.9073486328125e-06 elapsed_time : order_list_meta_photo_and_scores 5.9604644775390625e-06 ???????????????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.002028226852416992 elapsed_time : insert_dashboard_record_day_entry 0.022851943969726562 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20596211 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20596784 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20596785 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20596786 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20596786 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20596787 order by id desc limit 1 Qualite : 0.03486663619902253 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20596788_17-02-2025_07_35_27.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20596788 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20596788 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'17022025': {'nb_upload': 40, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1338117067, 1338117060, 1338117036, 1338117028, 1338117016, 1338117009, 1338116988, 1338116984, 1338116970, 1338116959] Looping around the photos to save general results len do output : 1 /20596788Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117067', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117060', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117036', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117028', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117016', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338117009', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116988', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116984', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116970', None, None, None, None, None, '2592316') ('3459', None, None, None, None, None, None, None, '2592316') ('3459', '20596788', '1338116959', None, None, None, None, None, '2592316') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013319253921508789 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.96074914932251 time spend to save output : 0.013565301895141602 total time spend for step 11 : 4.974314451217651 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 90.49user 86.45system 3:11.79elapsed 92%CPU (0avgtext+0avgdata 3632228maxresident)k 884528inputs+79552outputs (1079major+8608302minor)pagefaults 0swaps