python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1270047 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 30>10 please execute split_portfolio.py -i 20831712 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2615298'] with mtr_portfolio_ids : ['20831712'] and first list_photo_ids : [] new path : /proc/1270047/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.2823138236999512 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Wed Feb 26 09:24:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 1502 wait 20 seconds l 3637 free memory gpu now : 1502 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-26 09:24:52.982365: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-26 09:24:53.007045: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-26 09:24:53.009243: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f82f8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-26 09:24:53.009304: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-26 09:24:53.013410: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-26 09:24:53.183202: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2a4bd850 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-26 09:24:53.183249: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-26 09:24:53.184137: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-26 09:24:53.184456: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-26 09:24:53.186658: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-26 09:24:53.189332: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-26 09:24:53.189959: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-26 09:24:53.192863: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-26 09:24:53.193940: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-26 09:24:53.198866: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-26 09:24:53.200116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-26 09:24:53.200220: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-26 09:24:53.200940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-26 09:24:53.200957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-26 09:24:53.200966: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-26 09:24:53.202377: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 923 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-26 09:24:53.469137: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-26 09:24:53.469235: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-26 09:24:53.469256: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-26 09:24:53.469274: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-26 09:24:53.469293: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-26 09:24:53.469310: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-26 09:24:53.469328: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-26 09:24:53.469346: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-26 09:24:53.470262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-26 09:24:53.471239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-26 09:24:53.471272: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-26 09:24:53.471288: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-26 09:24:53.471302: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-26 09:24:53.471315: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-26 09:24:53.471329: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-26 09:24:53.471342: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-26 09:24:53.471356: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-26 09:24:53.472091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-26 09:24:53.472119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-26 09:24:53.472127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-26 09:24:53.472135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-26 09:24:53.472912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 923 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-26 09:25:01.127509: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-26 09:25:01.304658: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-26 09:25:02.816210: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 09:25:02.816292: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 09:25:02.822948: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 09:25:02.822983: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 09:25:02.829211: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 09:25:02.829267: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 09:25:02.860505: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 09:25:02.860551: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 09:25:02.866524: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:02.867074: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 371.59M (389637888 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:02.867601: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 334.43M (350674176 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:02.913301: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 09:25:02.913345: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-26 09:25:03.004576: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-02-26 09:25:03.019865: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.020590: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.051303: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.051865: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.055996: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.056544: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.072711: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.073272: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.106295: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.106826: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.111043: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.111566: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.112317: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.112843: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.123693: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.124239: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.124790: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.125321: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.191807: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.192338: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.219417: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.220048: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.220518: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.220984: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.225268: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.225749: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.226656: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.226673: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-02-26 09:25:03.251758: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.252362: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.253004: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.253631: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.254263: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-26 09:25:03.254876: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 412.88M (432930816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 Detection mask done ! Trying to reset tf kernel 1271662 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 309 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1502 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0006434917449951172 nb_pixel_total : 22585 time to create 1 rle with old method : 0.024479389190673828 length of segment : 280 time for calcul the mask position with numpy : 0.00030040740966796875 nb_pixel_total : 10848 time to create 1 rle with old method : 0.01214909553527832 length of segment : 169 time for calcul the mask position with numpy : 0.0005171298980712891 nb_pixel_total : 26282 time to create 1 rle with old method : 0.029244184494018555 length of segment : 200 time for calcul the mask position with numpy : 0.0030317306518554688 nb_pixel_total : 182535 time to create 1 rle with new method : 0.008430719375610352 length of segment : 515 time for calcul the mask position with numpy : 0.0002694129943847656 nb_pixel_total : 16628 time to create 1 rle with old method : 0.01886153221130371 length of segment : 106 time for calcul the mask position with numpy : 0.002148866653442383 nb_pixel_total : 88519 time to create 1 rle with old method : 0.09529733657836914 length of segment : 694 time for calcul the mask position with numpy : 0.0005199909210205078 nb_pixel_total : 24228 time to create 1 rle with old method : 0.028252363204956055 length of segment : 259 time for calcul the mask position with numpy : 0.00020194053649902344 nb_pixel_total : 6409 time to create 1 rle with old method : 0.007720232009887695 length of segment : 91 time for calcul the mask position with numpy : 0.016279220581054688 nb_pixel_total : 662544 time to create 1 rle with new method : 0.0827181339263916 length of segment : 1476 time for calcul the mask position with numpy : 0.0005753040313720703 nb_pixel_total : 21571 time to create 1 rle with old method : 0.027361392974853516 length of segment : 253 time for calcul the mask position with numpy : 0.005429983139038086 nb_pixel_total : 352493 time to create 1 rle with new method : 0.016280651092529297 length of segment : 1710 time for calcul the mask position with numpy : 7.343292236328125e-05 nb_pixel_total : 796 time to create 1 rle with old method : 0.0010135173797607422 length of segment : 31 time for calcul the mask position with numpy : 0.00014638900756835938 nb_pixel_total : 7917 time to create 1 rle with old method : 0.008886337280273438 length of segment : 103 time for calcul the mask position with numpy : 0.011151313781738281 nb_pixel_total : 653004 time to create 1 rle with new method : 0.032947540283203125 length of segment : 1607 time for calcul the mask position with numpy : 0.00022411346435546875 nb_pixel_total : 8251 time to create 1 rle with old method : 0.009132862091064453 length of segment : 104 time for calcul the mask position with numpy : 0.0005004405975341797 nb_pixel_total : 33196 time to create 1 rle with old method : 0.03720808029174805 length of segment : 236 time for calcul the mask position with numpy : 0.003565073013305664 nb_pixel_total : 295334 time to create 1 rle with new method : 0.01316690444946289 length of segment : 1023 time for calcul the mask position with numpy : 0.02596139907836914 nb_pixel_total : 1338241 time to create 1 rle with new method : 0.06148171424865723 length of segment : 1283 time for calcul the mask position with numpy : 0.0035185813903808594 nb_pixel_total : 255356 time to create 1 rle with new method : 0.009567975997924805 length of segment : 886 time for calcul the mask position with numpy : 0.0001881122589111328 nb_pixel_total : 8546 time to create 1 rle with old method : 0.009356021881103516 length of segment : 90 time for calcul the mask position with numpy : 0.003996372222900391 nb_pixel_total : 299780 time to create 1 rle with new method : 0.010089874267578125 length of segment : 1048 time for calcul the mask position with numpy : 0.03458905220031738 nb_pixel_total : 1406599 time to create 1 rle with new method : 0.31076550483703613 length of segment : 1695 time for calcul the mask position with numpy : 0.0003333091735839844 nb_pixel_total : 20714 time to create 1 rle with old method : 0.023917436599731445 length of segment : 149 time for calcul the mask position with numpy : 0.003262758255004883 nb_pixel_total : 287556 time to create 1 rle with new method : 0.007321834564208984 length of segment : 938 time for calcul the mask position with numpy : 0.0002911090850830078 nb_pixel_total : 5333 time to create 1 rle with old method : 0.009209871292114258 length of segment : 95 time for calcul the mask position with numpy : 0.03844904899597168 nb_pixel_total : 1470430 time to create 1 rle with new method : 0.22760224342346191 length of segment : 1544 time for calcul the mask position with numpy : 0.00034880638122558594 nb_pixel_total : 19463 time to create 1 rle with old method : 0.021741628646850586 length of segment : 168 time for calcul the mask position with numpy : 0.006168842315673828 nb_pixel_total : 390284 time to create 1 rle with new method : 0.013031959533691406 length of segment : 1386 time for calcul the mask position with numpy : 0.0001361370086669922 nb_pixel_total : 4085 time to create 1 rle with old method : 0.005143642425537109 length of segment : 117 time for calcul the mask position with numpy : 0.0033638477325439453 nb_pixel_total : 308355 time to create 1 rle with new method : 0.008300542831420898 length of segment : 944 time for calcul the mask position with numpy : 0.027196168899536133 nb_pixel_total : 1377521 time to create 1 rle with new method : 0.2849845886230469 length of segment : 1544 time for calcul the mask position with numpy : 0.003482341766357422 nb_pixel_total : 302846 time to create 1 rle with new method : 0.008497953414916992 length of segment : 956 time for calcul the mask position with numpy : 0.0006694793701171875 nb_pixel_total : 18727 time to create 1 rle with old method : 0.020184040069580078 length of segment : 281 time for calcul the mask position with numpy : 0.0009505748748779297 nb_pixel_total : 44820 time to create 1 rle with old method : 0.04912567138671875 length of segment : 244 time for calcul the mask position with numpy : 0.0005307197570800781 nb_pixel_total : 23320 time to create 1 rle with old method : 0.02992844581604004 length of segment : 249 time for calcul the mask position with numpy : 0.002554655075073242 nb_pixel_total : 87878 time to create 1 rle with old method : 0.09976530075073242 length of segment : 478 time for calcul the mask position with numpy : 0.004141807556152344 nb_pixel_total : 103858 time to create 1 rle with old method : 0.11660385131835938 length of segment : 585 time for calcul the mask position with numpy : 0.01149892807006836 nb_pixel_total : 576577 time to create 1 rle with new method : 0.10833907127380371 length of segment : 1419 time for calcul the mask position with numpy : 0.0011897087097167969 nb_pixel_total : 70113 time to create 1 rle with old method : 0.08123183250427246 length of segment : 289 time for calcul the mask position with numpy : 0.0007848739624023438 nb_pixel_total : 31095 time to create 1 rle with old method : 0.03564810752868652 length of segment : 439 time for calcul the mask position with numpy : 0.0005724430084228516 nb_pixel_total : 36481 time to create 1 rle with old method : 0.04275250434875488 length of segment : 245 time for calcul the mask position with numpy : 0.004005908966064453 nb_pixel_total : 296719 time to create 1 rle with new method : 0.011954784393310547 length of segment : 1119 time for calcul the mask position with numpy : 0.0014495849609375 nb_pixel_total : 66948 time to create 1 rle with old method : 0.07507443428039551 length of segment : 284 time for calcul the mask position with numpy : 0.000850677490234375 nb_pixel_total : 26330 time to create 1 rle with old method : 0.029623031616210938 length of segment : 297 time for calcul the mask position with numpy : 0.00041794776916503906 nb_pixel_total : 10549 time to create 1 rle with old method : 0.012502431869506836 length of segment : 215 time for calcul the mask position with numpy : 0.0192718505859375 nb_pixel_total : 1459306 time to create 1 rle with new method : 0.07292675971984863 length of segment : 1175 time for calcul the mask position with numpy : 0.0005102157592773438 nb_pixel_total : 28989 time to create 1 rle with old method : 0.03642082214355469 length of segment : 208 time for calcul the mask position with numpy : 0.0008535385131835938 nb_pixel_total : 64622 time to create 1 rle with old method : 0.0744476318359375 length of segment : 278 time for calcul the mask position with numpy : 0.003396749496459961 nb_pixel_total : 298392 time to create 1 rle with new method : 0.007897615432739258 length of segment : 1009 time for calcul the mask position with numpy : 0.0022683143615722656 nb_pixel_total : 183316 time to create 1 rle with new method : 0.006062984466552734 length of segment : 1163 time for calcul the mask position with numpy : 0.0005047321319580078 nb_pixel_total : 12803 time to create 1 rle with old method : 0.020912885665893555 length of segment : 183 time for calcul the mask position with numpy : 0.012048482894897461 nb_pixel_total : 450567 time to create 1 rle with new method : 0.12595462799072266 length of segment : 1089 time for calcul the mask position with numpy : 0.004482269287109375 nb_pixel_total : 379937 time to create 1 rle with new method : 0.011593103408813477 length of segment : 1711 time for calcul the mask position with numpy : 6.532669067382812e-05 nb_pixel_total : 948 time to create 1 rle with old method : 0.0012531280517578125 length of segment : 35 time for calcul the mask position with numpy : 0.000255584716796875 nb_pixel_total : 13295 time to create 1 rle with old method : 0.015691757202148438 length of segment : 243 time spent for convertir_results : 7.48210072517395 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 55 chid ids of type : 3663 Number RLEs to save : 34938 save missing photos in datou_result : time spend for datou_step_exec : 48.40295886993408 time spend to save output : 1.968660593032837 total time spend for step 1 : 50.37161946296692 step2:crop_condition Wed Feb 26 09:25:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 55 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1339688545, 1339688492, 1339688489, 1339688440, 1339688434, 1339688397, 1339688395, 1339688356, 1339688354, 1339688319] Looping around the photos to save general results len do output : 19 /-3687283155Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283154Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283153Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283151Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283158Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283177Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283194Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283152Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283191Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283189Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283175Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283179Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283193Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283197Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283185Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283157Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283163Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283173Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3687283198Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688545', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688492', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688489', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688440', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688434', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688397', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688395', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688356', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688354', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688319', None, None, None, None, None, '2615298') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 67 time used for this insertion : 0.022825241088867188 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.308698892593384 time spend to save output : 0.023693323135375977 total time spend for step 2 : 4.33239221572876 step3:thcl Wed Feb 26 09:25:24 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.0584256649017334 time to convert the images to numpy array : 0.009200096130371094 time to import caffe and check if the image exist : 0.06241464614868164 time to convert the images to numpy array : 0.006788969039916992 time to import caffe and check if the image exist : 0.06308221817016602 time to convert the images to numpy array : 0.007451295852661133 time to import caffe and check if the image exist : 0.05966925621032715 time to convert the images to numpy array : 0.01381230354309082 time to import caffe and check if the image exist : 0.06237196922302246 time to convert the images to numpy array : 0.012758970260620117 time to import caffe and check if the image exist : 0.06014370918273926 time to convert the images to numpy array : 0.015271902084350586 time to import caffe and check if the image exist : 0.05792999267578125 time to convert the images to numpy array : 0.017473936080932617 time to import caffe and check if the image exist : 0.05936884880065918 time to convert the images to numpy array : 0.01910996437072754 time to import caffe and check if the image exist : 0.05639457702636719 time to convert the images to numpy array : 0.023358583450317383 time to import caffe and check if the image exist : 0.05835580825805664 time to convert the images to numpy array : 0.03402137756347656 total time to convert the images to numpy array : 0.2785336971282959 list photo_ids error: [] list photo_ids correct : [-3687283175, -3687283179, -3687283163, -3687283173, -3687283158, -3687283177, -3687283153, -3687283151, -3687283185, -3687283157, -3687283194, -3687283152, -3687283198, -3687283193, -3687283197, -3687283191, -3687283189, -3687283155, -3687283154] number of photos to traite : 19 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1502 wait 20 seconds l 3637 free memory gpu now : 1502 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1504 wait 20 seconds l 3637 free memory gpu now : 1504 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.1582934856414795 time used to do the prediction : 0.18008136749267578 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1502 wait 20 seconds l 3637 free memory gpu now : 1502 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1502 wait 20 seconds l 3637 free memory gpu now : 1502 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.16572189331054688 time used to do the prediction : 0.07546758651733398 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 3.0994415283203125e-05 save missing photos in datou_result : time spend for datou_step_exec : 87.52714252471924 time spend to save output : 0.00029468536376953125 total time spend for step 3 : 87.52743721008301 step4:merge_mask_thcl_custom Wed Feb 26 09:26:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 55 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1339688545, 1339688492, 1339688489, 1339688440, 1339688434, 1339688397, 1339688395, 1339688356, 1339688354, 1339688319] Looping around the photos to save general results len do output : 10 /1339688545Didn't retrieve data .Didn't retrieve data . /1339688492Didn't retrieve data .Didn't retrieve data . /1339688489Didn't retrieve data .Didn't retrieve data . /1339688440Didn't retrieve data .Didn't retrieve data . /1339688434Didn't retrieve data .Didn't retrieve data . /1339688397Didn't retrieve data .Didn't retrieve data . /1339688395Didn't retrieve data .Didn't retrieve data . /1339688356Didn't retrieve data .Didn't retrieve data . /1339688354Didn't retrieve data .Didn't retrieve data . /1339688319Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688545', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688492', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688489', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688440', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688434', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688397', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688395', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688356', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688354', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688319', None, None, None, None, None, '2615298') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01446080207824707 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.04800844192504883 time spend to save output : 0.014977455139160156 total time spend for step 4 : 0.06298589706420898 step5:rle_unique_nms_with_priority Wed Feb 26 09:26:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 6 nb_hashtags : 4 time to prepare the origin masks : 1.4819207191467285 time for calcul the mask position with numpy : 0.3001534938812256 nb_pixel_total : 1727047 time to create 1 rle with new method : 0.26357483863830566 time for calcul the mask position with numpy : 0.006741523742675781 nb_pixel_total : 10804 time to create 1 rle with old method : 0.012538433074951172 time for calcul the mask position with numpy : 0.007169961929321289 nb_pixel_total : 182499 time to create 1 rle with new method : 0.09438180923461914 time for calcul the mask position with numpy : 0.006608724594116211 nb_pixel_total : 87944 time to create 1 rle with old method : 0.09932518005371094 time for calcul the mask position with numpy : 0.006887912750244141 nb_pixel_total : 16587 time to create 1 rle with old method : 0.01947474479675293 time for calcul the mask position with numpy : 0.00817728042602539 nb_pixel_total : 26212 time to create 1 rle with old method : 0.030350446701049805 time for calcul the mask position with numpy : 0.006775379180908203 nb_pixel_total : 22507 time to create 1 rle with old method : 0.026216745376586914 create new chi : 0.9064433574676514 time to delete rle : 0.023663759231567383 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 5034 TO DO : save crop sub photo not yet done ! save time : 0.32321739196777344 nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 1.134873628616333 time for calcul the mask position with numpy : 0.05249357223510742 nb_pixel_total : 1184481 time to create 1 rle with new method : 0.261486291885376 time for calcul the mask position with numpy : 0.006941795349121094 nb_pixel_total : 10995 time to create 1 rle with old method : 0.013031244277954102 time for calcul the mask position with numpy : 0.0068781375885009766 nb_pixel_total : 788 time to create 1 rle with old method : 0.0014710426330566406 time for calcul the mask position with numpy : 0.011912107467651367 nb_pixel_total : 169233 time to create 1 rle with new method : 0.2751195430755615 time for calcul the mask position with numpy : 0.006773233413696289 nb_pixel_total : 6387 time to create 1 rle with old method : 0.007441043853759766 time for calcul the mask position with numpy : 0.007154941558837891 nb_pixel_total : 21473 time to create 1 rle with old method : 0.02467179298400879 time for calcul the mask position with numpy : 0.006844758987426758 nb_pixel_total : 7895 time to create 1 rle with old method : 0.008940696716308594 time for calcul the mask position with numpy : 0.013297080993652344 nb_pixel_total : 672348 time to create 1 rle with new method : 0.03054046630859375 create new chi : 0.7457232475280762 time to delete rle : 0.0006933212280273438 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6387 TO DO : save crop sub photo not yet done ! save time : 0.3913106918334961 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.42101192474365234 time for calcul the mask position with numpy : 0.018360614776611328 nb_pixel_total : 1273907 time to create 1 rle with new method : 0.09270715713500977 time for calcul the mask position with numpy : 0.0066874027252197266 nb_pixel_total : 30848 time to create 1 rle with old method : 0.033243656158447266 time for calcul the mask position with numpy : 0.006616353988647461 nb_pixel_total : 37988 time to create 1 rle with old method : 0.039546966552734375 time for calcul the mask position with numpy : 0.0065081119537353516 nb_pixel_total : 8141 time to create 1 rle with old method : 0.00869894027709961 time for calcul the mask position with numpy : 0.013152599334716797 nb_pixel_total : 722716 time to create 1 rle with new method : 0.03522300720214844 create new chi : 0.26149487495422363 time to delete rle : 0.0005218982696533203 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 4536 TO DO : save crop sub photo not yet done ! save time : 0.29302024841308594 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.47510337829589844 time for calcul the mask position with numpy : 0.010848760604858398 nb_pixel_total : 570510 time to create 1 rle with new method : 0.20110321044921875 time for calcul the mask position with numpy : 0.00715947151184082 nb_pixel_total : 131044 time to create 1 rle with old method : 0.15175938606262207 time for calcul the mask position with numpy : 0.006372928619384766 nb_pixel_total : 8499 time to create 1 rle with old method : 0.00953817367553711 time for calcul the mask position with numpy : 0.007197856903076172 nb_pixel_total : 28687 time to create 1 rle with old method : 0.03373265266418457 time for calcul the mask position with numpy : 0.05061197280883789 nb_pixel_total : 1334860 time to create 1 rle with new method : 0.1588292121887207 create new chi : 0.6530735492706299 time to delete rle : 0.000614166259765625 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 5754 TO DO : save crop sub photo not yet done ! save time : 0.3596186637878418 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 0.2970764636993408 time for calcul the mask position with numpy : 0.011361122131347656 nb_pixel_total : 573680 time to create 1 rle with new method : 0.09548139572143555 time for calcul the mask position with numpy : 0.007126331329345703 nb_pixel_total : 95492 time to create 1 rle with old method : 0.10963869094848633 time for calcul the mask position with numpy : 0.022016286849975586 nb_pixel_total : 1404428 time to create 1 rle with new method : 0.11068201065063477 create new chi : 0.3640103340148926 time to delete rle : 0.0006258487701416016 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 5211 TO DO : save crop sub photo not yet done ! save time : 0.32201147079467773 nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.7210707664489746 time for calcul the mask position with numpy : 0.00877690315246582 nb_pixel_total : 266740 time to create 1 rle with new method : 0.08952593803405762 time for calcul the mask position with numpy : 0.00713658332824707 nb_pixel_total : 47739 time to create 1 rle with old method : 0.054913997650146484 time for calcul the mask position with numpy : 0.0072479248046875 nb_pixel_total : 85888 time to create 1 rle with old method : 0.09958982467651367 time for calcul the mask position with numpy : 0.34643054008483887 nb_pixel_total : 1653767 time to create 1 rle with new method : 0.10638689994812012 time for calcul the mask position with numpy : 0.007277727127075195 nb_pixel_total : 19466 time to create 1 rle with old method : 0.02154994010925293 create new chi : 0.7631046772003174 time to delete rle : 0.0007791519165039062 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 5284 TO DO : save crop sub photo not yet done ! save time : 0.3276245594024658 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 0.26336121559143066 time for calcul the mask position with numpy : 0.012654304504394531 nb_pixel_total : 546806 time to create 1 rle with new method : 0.31888318061828613 time for calcul the mask position with numpy : 0.007635831832885742 nb_pixel_total : 152378 time to create 1 rle with new method : 0.444019079208374 time for calcul the mask position with numpy : 0.02154994010925293 nb_pixel_total : 1374416 time to create 1 rle with new method : 0.08025431632995605 create new chi : 0.9040660858154297 time to delete rle : 0.0006854534149169922 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 5198 TO DO : save crop sub photo not yet done ! save time : 0.3214578628540039 nb_obj : 11 nb_hashtags : 4 time to prepare the origin masks : 2.33866024017334 time for calcul the mask position with numpy : 0.046122074127197266 nb_pixel_total : 1177415 time to create 1 rle with new method : 0.1053311824798584 time for calcul the mask position with numpy : 0.007955789566040039 nb_pixel_total : 77803 time to create 1 rle with old method : 0.09840869903564453 time for calcul the mask position with numpy : 0.007541179656982422 nb_pixel_total : 4287 time to create 1 rle with old method : 0.0054891109466552734 time for calcul the mask position with numpy : 0.0076601505279541016 nb_pixel_total : 8290 time to create 1 rle with old method : 0.010958194732666016 time for calcul the mask position with numpy : 0.010659456253051758 nb_pixel_total : 522539 time to create 1 rle with new method : 0.09942770004272461 time for calcul the mask position with numpy : 0.0066089630126953125 nb_pixel_total : 7534 time to create 1 rle with old method : 0.010047674179077148 time for calcul the mask position with numpy : 0.006761074066162109 nb_pixel_total : 23342 time to create 1 rle with old method : 0.02849435806274414 time for calcul the mask position with numpy : 0.006728410720825195 nb_pixel_total : 69855 time to create 1 rle with old method : 0.08161306381225586 time for calcul the mask position with numpy : 0.006446123123168945 nb_pixel_total : 31702 time to create 1 rle with old method : 0.03537273406982422 time for calcul the mask position with numpy : 0.006962776184082031 nb_pixel_total : 87520 time to create 1 rle with old method : 0.1029200553894043 time for calcul the mask position with numpy : 0.00924372673034668 nb_pixel_total : 44669 time to create 1 rle with old method : 0.05014753341674805 time for calcul the mask position with numpy : 0.006991386413574219 nb_pixel_total : 18644 time to create 1 rle with old method : 0.023766040802001953 create new chi : 0.7933022975921631 time to delete rle : 0.0009446144104003906 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 8874 TO DO : save crop sub photo not yet done ! save time : 0.5100488662719727 nb_obj : 6 nb_hashtags : 3 time to prepare the origin masks : 0.4308586120605469 time for calcul the mask position with numpy : 0.009712696075439453 nb_pixel_total : 502441 time to create 1 rle with new method : 0.03212761878967285 time for calcul the mask position with numpy : 0.006955146789550781 nb_pixel_total : 32547 time to create 1 rle with old method : 0.03662228584289551 time for calcul the mask position with numpy : 0.007066965103149414 nb_pixel_total : 63167 time to create 1 rle with old method : 0.0758054256439209 time for calcul the mask position with numpy : 0.00677180290222168 nb_pixel_total : 2946 time to create 1 rle with old method : 0.003568887710571289 time for calcul the mask position with numpy : 0.016227006912231445 nb_pixel_total : 1435737 time to create 1 rle with new method : 0.032317399978637695 time for calcul the mask position with numpy : 0.0067119598388671875 nb_pixel_total : 10531 time to create 1 rle with old method : 0.011564493179321289 time for calcul the mask position with numpy : 0.006937503814697266 nb_pixel_total : 26231 time to create 1 rle with old method : 0.029894113540649414 create new chi : 0.29044294357299805 time to delete rle : 0.0010082721710205078 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 6259 TO DO : save crop sub photo not yet done ! save time : 0.36787986755371094 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.52825927734375 time for calcul the mask position with numpy : 0.016529321670532227 nb_pixel_total : 1386847 time to create 1 rle with new method : 0.031429290771484375 time for calcul the mask position with numpy : 0.00651240348815918 nb_pixel_total : 13237 time to create 1 rle with old method : 0.01524972915649414 time for calcul the mask position with numpy : 0.007730722427368164 nb_pixel_total : 208691 time to create 1 rle with new method : 0.03011155128479004 time for calcul the mask position with numpy : 0.0091400146484375 nb_pixel_total : 452065 time to create 1 rle with new method : 0.029134750366210938 time for calcul the mask position with numpy : 0.006651639938354492 nb_pixel_total : 12760 time to create 1 rle with old method : 0.02036142349243164 create new chi : 0.17380809783935547 time to delete rle : 0.0008976459503173828 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 5675 TO DO : save crop sub photo not yet done ! save time : 0.3558533191680908 map_output_result : {1339688545: (0.0, 'Should be the crop_list due to order', 0), 1339688492: (0.0, 'Should be the crop_list due to order', 0), 1339688489: (0.0, 'Should be the crop_list due to order', 0), 1339688440: (0.0, 'Should be the crop_list due to order', 0), 1339688434: (0.0, 'Should be the crop_list due to order', 0), 1339688397: (0.0, 'Should be the crop_list due to order', 0), 1339688395: (0.0, 'Should be the crop_list due to order', 0), 1339688356: (0.0, 'Should be the crop_list due to order', 0), 1339688354: (0.0, 'Should be the crop_list due to order', 0), 1339688319: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1339688545, 1339688492, 1339688489, 1339688440, 1339688434, 1339688397, 1339688395, 1339688356, 1339688354, 1339688319] Looping around the photos to save general results len do output : 10 /1339688545.Didn't retrieve data . /1339688492.Didn't retrieve data . /1339688489.Didn't retrieve data . /1339688440.Didn't retrieve data . /1339688434.Didn't retrieve data . /1339688397.Didn't retrieve data . /1339688395.Didn't retrieve data . /1339688356.Didn't retrieve data . /1339688354.Didn't retrieve data . /1339688319.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688545', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688492', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688489', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688440', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688434', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688397', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688395', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688356', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688354', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688319', None, None, None, None, None, '2615298') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013782739639282227 save_final save missing photos in datou_result : time spend for datou_step_exec : 17.881993055343628 time spend to save output : 0.014307498931884766 total time spend for step 5 : 17.896300554275513 step6:crop_condition Wed Feb 26 09:27:10 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 60 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740558431_1270047 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.649507999420166 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740558433_1270047 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.0456833839416504 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740558436_1270047 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9124176502227783 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740558437_1270047 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6113579273223877 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1339688545, 1339688492, 1339688489, 1339688440, 1339688434, 1339688397, 1339688395, 1339688356, 1339688354, 1339688319] Looping around the photos to save general results len do output : 10 /1339695212Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339695213Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339695214Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339695215Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339695216Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339695220Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339695221Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339695222Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339695225Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339695227Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688545', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688492', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688489', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688440', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688434', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688397', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688395', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688356', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688354', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688319', None, None, None, None, None, '2615298') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 40 time used for this insertion : 0.022367477416992188 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.537235260009766 time spend to save output : 0.022949695587158203 total time spend for step 6 : 7.560184955596924 step7:ventilate_hashtags_in_portfolio Wed Feb 26 09:27:18 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20831712 get user id for portfolio 20831712 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20831712 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','Teint_Dans_La_Masse','cartonnette','Carton_gris','flou','Carton_brun','papier','kraft','autre_refus','mal_croppe','metal','plastique')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20831712 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','Teint_Dans_La_Masse','cartonnette','Carton_gris','flou','Carton_brun','papier','kraft','autre_refus','mal_croppe','metal','plastique')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20831712 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','Teint_Dans_La_Masse','cartonnette','Carton_gris','flou','Carton_brun','papier','kraft','autre_refus','mal_croppe','metal','plastique')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20832376,20832377,20832378,20832379,20832380,20832381,20832382,20832383,20832384,20832385,20832386,20832387?tags=environnement,Teint_Dans_La_Masse,cartonnette,Carton_gris,flou,Carton_brun,papier,kraft,autre_refus,mal_croppe,metal,plastique Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1339688545, 1339688492, 1339688489, 1339688440, 1339688434, 1339688397, 1339688395, 1339688356, 1339688354, 1339688319] Looping around the photos to save general results len do output : 1 /20831712. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688545', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688492', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688489', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688440', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688434', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688397', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688395', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688356', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688354', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688319', None, None, None, None, None, '2615298') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013643741607666016 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.6647446155548096 time spend to save output : 0.014045000076293945 total time spend for step 7 : 0.6787896156311035 step8:final Wed Feb 26 09:27:18 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1339688545: ('0.05143760370422941',), 1339688492: ('0.05143760370422941',), 1339688489: ('0.05143760370422941',), 1339688440: ('0.05143760370422941',), 1339688434: ('0.05143760370422941',), 1339688397: ('0.05143760370422941',), 1339688395: ('0.05143760370422941',), 1339688356: ('0.05143760370422941',), 1339688354: ('0.05143760370422941',), 1339688319: ('0.05143760370422941',)} new output for save of step final : {1339688545: ('0.05143760370422941',), 1339688492: ('0.05143760370422941',), 1339688489: ('0.05143760370422941',), 1339688440: ('0.05143760370422941',), 1339688434: ('0.05143760370422941',), 1339688397: ('0.05143760370422941',), 1339688395: ('0.05143760370422941',), 1339688356: ('0.05143760370422941',), 1339688354: ('0.05143760370422941',), 1339688319: ('0.05143760370422941',)} [1339688545, 1339688492, 1339688489, 1339688440, 1339688434, 1339688397, 1339688395, 1339688356, 1339688354, 1339688319] Looping around the photos to save general results len do output : 10 /1339688545.Didn't retrieve data . /1339688492.Didn't retrieve data . /1339688489.Didn't retrieve data . /1339688440.Didn't retrieve data . /1339688434.Didn't retrieve data . /1339688397.Didn't retrieve data . /1339688395.Didn't retrieve data . /1339688356.Didn't retrieve data . /1339688354.Didn't retrieve data . /1339688319.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688545', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688492', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688489', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688440', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688434', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688397', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688395', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688356', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688354', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688319', None, None, None, None, None, '2615298') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01415562629699707 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10457491874694824 time spend to save output : 0.014702796936035156 total time spend for step 8 : 0.1192777156829834 step9:velours_tree Wed Feb 26 09:27:18 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.03056025505065918 time spend to save output : 3.2901763916015625e-05 total time spend for step 9 : 0.030593156814575195 step10:send_mail_cod Wed Feb 26 09:27:18 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20831712_26-02-2025_09_27_18.pdf 20832378 imagette208323781740558438 20832380 imagette208323801740558438 20832383 imagette208323831740558438 20832384 change filename to text .change filename to text .change filename to text .imagette208323841740558438 20832385 imagette208323851740558439 20832386 imagette208323861740558439 20832387 imagette208323871740558439 20832377 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208323771740558439 20832379 change filename to text .imagette208323791740558439 20832381 change filename to text .imagette208323811740558439 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20831712 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20832376,20832377,20832378,20832379,20832380,20832381,20832382,20832383,20832384,20832385,20832386,20832387?tags=environnement,Teint_Dans_La_Masse,cartonnette,Carton_gris,flou,Carton_brun,papier,kraft,autre_refus,mal_croppe,metal,plastique your option no_mail is active, we will not send the real mail to your client args[1339688545] : ('0.05143760370422941',) no score found for photo 1339688545 We are sending mail with results at report@fotonower.com args[1339688492] : ('0.05143760370422941',) no score found for photo 1339688492 We are sending mail with results at report@fotonower.com args[1339688489] : ('0.05143760370422941',) no score found for photo 1339688489 We are sending mail with results at report@fotonower.com args[1339688440] : ('0.05143760370422941',) no score found for photo 1339688440 We are sending mail with results at report@fotonower.com args[1339688434] : ('0.05143760370422941',) no score found for photo 1339688434 We are sending mail with results at report@fotonower.com args[1339688397] : ('0.05143760370422941',) no score found for photo 1339688397 We are sending mail with results at report@fotonower.com args[1339688395] : ('0.05143760370422941',) no score found for photo 1339688395 We are sending mail with results at report@fotonower.com args[1339688356] : ('0.05143760370422941',) no score found for photo 1339688356 We are sending mail with results at report@fotonower.com args[1339688354] : ('0.05143760370422941',) no score found for photo 1339688354 We are sending mail with results at report@fotonower.com args[1339688319] : ('0.05143760370422941',) no score found for photo 1339688319 We are sending mail with results at report@fotonower.com refus_total : 0.05143760370422941 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20831712 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1339687234,1339688043,1339688545,1339688492,1339688489,1339688440,1339688434,1339688397,1339688395,1339688356,1339688354,1339688319,1339688317,1339688268,1339688253,1339688045,1339686715,1339687581,1339687580,1339687474) Found this number of photos: 20 begin to download photo : 1339687234 begin to download photo : 1339688440 begin to download photo : 1339688354 begin to download photo : 1339688045 download finish for photo 1339688440 begin to download photo : 1339688434 download finish for photo 1339688354 begin to download photo : 1339688319 download finish for photo 1339687234 begin to download photo : 1339688043 download finish for photo 1339688045 begin to download photo : 1339686715 download finish for photo 1339688319 begin to download photo : 1339688317 download finish for photo 1339688434 begin to download photo : 1339688397 download finish for photo 1339688043 begin to download photo : 1339688545 download finish for photo 1339688397 begin to download photo : 1339688395 download finish for photo 1339688317 begin to download photo : 1339688268 download finish for photo 1339686715 begin to download photo : 1339687581 download finish for photo 1339688395 begin to download photo : 1339688356 download finish for photo 1339688545 begin to download photo : 1339688492 download finish for photo 1339687581 begin to download photo : 1339687580 download finish for photo 1339688356 download finish for photo 1339688268 begin to download photo : 1339688253 download finish for photo 1339688492 begin to download photo : 1339688489 download finish for photo 1339687580 begin to download photo : 1339687474 download finish for photo 1339688489 download finish for photo 1339687474 download finish for photo 1339688253 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20831712_26-02-2025_09_27_18.pdf results_Auto_P20831712_26-02-2025_09_27_18.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20831712_26-02-2025_09_27_18.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20831712','results_Auto_P20831712_26-02-2025_09_27_18.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20831712_26-02-2025_09_27_18.pdf','pdf','','0.22','0.05143760370422941') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1339688545, 1339688492, 1339688489, 1339688440, 1339688434, 1339688397, 1339688395, 1339688356, 1339688354, 1339688319] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688545', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688492', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688489', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688440', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688434', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688397', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688395', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688356', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688354', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688319', None, None, None, None, None, '2615298') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.0135955810546875 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.013732433319092 time spend to save output : 0.014241218566894531 total time spend for step 10 : 3.0279736518859863 step11:split_time_score Wed Feb 26 09:27:21 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('08', 30),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 26022025 20831712 Nombre de photos uploadées : 30 / 23040 (0%) 26022025 20831712 Nombre de photos taguées (types de déchets): 0 / 30 (0%) 26022025 20831712 Nombre de photos taguées (volume) : 0 / 30 (0%) elapsed_time : load_data_split_time_score 3.337860107421875e-06 elapsed_time : order_list_meta_photo_and_scores 6.198883056640625e-06 ?????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0012269020080566406 elapsed_time : insert_dashboard_record_day_entry 0.025458574295043945 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.02254081664272345 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20827856_26-02-2025_07_51_37.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20827856 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20827856 AND mptpi.`type`=3726 To do Qualite : 0.017155428794724092 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20827857_26-02-2025_07_43_40.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20827857 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20827857 AND mptpi.`type`=3726 To do Qualite : 0.030893797764585375 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20829133_26-02-2025_08_11_55.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20829133 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20829133 AND mptpi.`type`=3726 To do Qualite : 0.04827767634589064 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20829134_26-02-2025_08_17_37.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20829134 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20829134 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20831711 order by id desc limit 1 Qualite : 0.05143760370422941 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20831712_26-02-2025_09_27_18.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20831712 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20831712 AND mptpi.`type`=3726 To do Qualite : 0.032421957006809145 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20831713_26-02-2025_09_25_21.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20831713 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20831713 AND mptpi.`type`=3726 To do Qualite : 0.051330577474808986 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20831714_26-02-2025_09_18_47.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20831714 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20831714 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'26022025': {'nb_upload': 30, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1339688545, 1339688492, 1339688489, 1339688440, 1339688434, 1339688397, 1339688395, 1339688356, 1339688354, 1339688319] Looping around the photos to save general results len do output : 1 /20831712Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688545', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688492', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688489', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688440', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688434', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688397', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688395', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688356', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688354', None, None, None, None, None, '2615298') ('3459', None, None, None, None, None, None, None, '2615298') ('3459', '20831712', '1339688319', None, None, None, None, None, '2615298') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014157772064208984 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.4643702507019043 time spend to save output : 0.014434099197387695 total time spend for step 11 : 1.478804349899292 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 38.20user 28.31system 2:59.07elapsed 37%CPU (0avgtext+0avgdata 2638780maxresident)k 976inputs+27400outputs (31major+2080163minor)pagefaults 0swaps