python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3785188 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 22>10 please execute split_portfolio.py -i 20915779 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2622650'] with mtr_portfolio_ids : ['20915779'] and first list_photo_ids : [] new path : /proc/3785188/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.2072596549987793 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 28 08:10:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-28 08:10:32.928117: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-28 08:10:32.955361: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-28 08:10:32.957821: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f96d8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-28 08:10:32.957901: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-28 08:10:32.962260: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-28 08:10:33.229836: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x779a590 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-28 08:10:33.229908: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-28 08:10:33.231160: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 08:10:33.231877: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 08:10:33.235410: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 08:10:33.238230: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 08:10:33.239064: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 08:10:33.242419: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 08:10:33.244101: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 08:10:33.248890: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 08:10:33.250091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 08:10:33.250158: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 08:10:33.250808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-28 08:10:33.250829: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-28 08:10:33.250840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-28 08:10:33.251982: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4631 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-28 08:10:33.636863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 08:10:33.636999: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 08:10:33.637024: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 08:10:33.637045: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 08:10:33.637064: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 08:10:33.637084: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 08:10:33.637103: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 08:10:33.637123: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 08:10:33.638361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 08:10:33.639741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 08:10:33.639796: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 08:10:33.639817: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 08:10:33.639836: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 08:10:33.639855: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 08:10:33.639873: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 08:10:33.639891: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 08:10:33.639911: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 08:10:33.641143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 08:10:33.641181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-28 08:10:33.641192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-28 08:10:33.641202: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-28 08:10:33.642500: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4631 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-28 08:10:43.012507: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 08:10:43.215823: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 08:10:44.601871: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 145.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 13 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 Detection mask done ! Trying to reset tf kernel 3785972 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 202 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5083 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.004465818405151367 nb_pixel_total : 124460 time to create 1 rle with old method : 0.14292263984680176 length of segment : 484 time for calcul the mask position with numpy : 0.00026488304138183594 nb_pixel_total : 6132 time to create 1 rle with old method : 0.008400440216064453 length of segment : 67 time for calcul the mask position with numpy : 0.00045609474182128906 nb_pixel_total : 12434 time to create 1 rle with old method : 0.015706300735473633 length of segment : 138 time for calcul the mask position with numpy : 0.0004584789276123047 nb_pixel_total : 9195 time to create 1 rle with old method : 0.011571884155273438 length of segment : 169 time for calcul the mask position with numpy : 0.0003139972686767578 nb_pixel_total : 6713 time to create 1 rle with old method : 0.009011268615722656 length of segment : 76 time for calcul the mask position with numpy : 0.0004413127899169922 nb_pixel_total : 10780 time to create 1 rle with old method : 0.016744136810302734 length of segment : 116 time for calcul the mask position with numpy : 0.0010523796081542969 nb_pixel_total : 29196 time to create 1 rle with old method : 0.045975685119628906 length of segment : 282 time for calcul the mask position with numpy : 0.010284185409545898 nb_pixel_total : 403476 time to create 1 rle with new method : 0.043562889099121094 length of segment : 958 time for calcul the mask position with numpy : 0.002144336700439453 nb_pixel_total : 90965 time to create 1 rle with old method : 0.10302019119262695 length of segment : 725 time for calcul the mask position with numpy : 0.0002110004425048828 nb_pixel_total : 7573 time to create 1 rle with old method : 0.008994579315185547 length of segment : 111 time for calcul the mask position with numpy : 0.016561508178710938 nb_pixel_total : 853520 time to create 1 rle with new method : 0.4059152603149414 length of segment : 1616 time for calcul the mask position with numpy : 0.0038039684295654297 nb_pixel_total : 284215 time to create 1 rle with new method : 0.008757829666137695 length of segment : 608 time for calcul the mask position with numpy : 0.00026106834411621094 nb_pixel_total : 11537 time to create 1 rle with old method : 0.01400899887084961 length of segment : 185 time for calcul the mask position with numpy : 0.000370025634765625 nb_pixel_total : 21931 time to create 1 rle with old method : 0.025584697723388672 length of segment : 171 time for calcul the mask position with numpy : 0.000308990478515625 nb_pixel_total : 13305 time to create 1 rle with old method : 0.015929222106933594 length of segment : 160 time for calcul the mask position with numpy : 0.0034923553466796875 nb_pixel_total : 293640 time to create 1 rle with new method : 0.007862091064453125 length of segment : 907 time for calcul the mask position with numpy : 0.0004067420959472656 nb_pixel_total : 24231 time to create 1 rle with old method : 0.02875995635986328 length of segment : 192 time for calcul the mask position with numpy : 0.0008654594421386719 nb_pixel_total : 23754 time to create 1 rle with old method : 0.028701305389404297 length of segment : 299 time for calcul the mask position with numpy : 0.00033974647521972656 nb_pixel_total : 16397 time to create 1 rle with old method : 0.019115209579467773 length of segment : 233 time for calcul the mask position with numpy : 0.013287782669067383 nb_pixel_total : 818225 time to create 1 rle with new method : 0.022801876068115234 length of segment : 1444 time for calcul the mask position with numpy : 0.0008814334869384766 nb_pixel_total : 54032 time to create 1 rle with old method : 0.06077885627746582 length of segment : 329 time for calcul the mask position with numpy : 0.007842063903808594 nb_pixel_total : 294953 time to create 1 rle with new method : 0.012931108474731445 length of segment : 929 time for calcul the mask position with numpy : 0.0005095005035400391 nb_pixel_total : 16650 time to create 1 rle with old method : 0.018959999084472656 length of segment : 155 time for calcul the mask position with numpy : 0.00013780593872070312 nb_pixel_total : 1538 time to create 1 rle with old method : 0.00196075439453125 length of segment : 60 time for calcul the mask position with numpy : 0.0002357959747314453 nb_pixel_total : 5643 time to create 1 rle with old method : 0.0070683956146240234 length of segment : 95 time for calcul the mask position with numpy : 0.01573777198791504 nb_pixel_total : 879338 time to create 1 rle with new method : 0.03327369689941406 length of segment : 1232 time for calcul the mask position with numpy : 0.0001983642578125 nb_pixel_total : 7910 time to create 1 rle with old method : 0.009895801544189453 length of segment : 93 time for calcul the mask position with numpy : 0.0011875629425048828 nb_pixel_total : 75149 time to create 1 rle with old method : 0.08665704727172852 length of segment : 349 time for calcul the mask position with numpy : 0.0013539791107177734 nb_pixel_total : 59173 time to create 1 rle with old method : 0.0681467056274414 length of segment : 293 time for calcul the mask position with numpy : 0.0018568038940429688 nb_pixel_total : 77625 time to create 1 rle with old method : 0.08803391456604004 length of segment : 313 time for calcul the mask position with numpy : 0.0005025863647460938 nb_pixel_total : 16908 time to create 1 rle with old method : 0.020051002502441406 length of segment : 112 time for calcul the mask position with numpy : 0.008891582489013672 nb_pixel_total : 356082 time to create 1 rle with new method : 0.015677928924560547 length of segment : 930 time for calcul the mask position with numpy : 0.00020885467529296875 nb_pixel_total : 8152 time to create 1 rle with old method : 0.009490251541137695 length of segment : 133 time for calcul the mask position with numpy : 0.002376079559326172 nb_pixel_total : 125065 time to create 1 rle with old method : 0.14034152030944824 length of segment : 603 time for calcul the mask position with numpy : 0.00041961669921875 nb_pixel_total : 13364 time to create 1 rle with old method : 0.016616106033325195 length of segment : 164 time for calcul the mask position with numpy : 0.0002646446228027344 nb_pixel_total : 9507 time to create 1 rle with old method : 0.011816978454589844 length of segment : 104 time for calcul the mask position with numpy : 0.004956245422363281 nb_pixel_total : 357076 time to create 1 rle with new method : 0.012908697128295898 length of segment : 1083 time for calcul the mask position with numpy : 0.0005354881286621094 nb_pixel_total : 10402 time to create 1 rle with old method : 0.01698756217956543 length of segment : 128 time for calcul the mask position with numpy : 0.004961967468261719 nb_pixel_total : 257510 time to create 1 rle with new method : 0.008694648742675781 length of segment : 883 time for calcul the mask position with numpy : 0.014190673828125 nb_pixel_total : 938595 time to create 1 rle with new method : 0.029397249221801758 length of segment : 2171 time for calcul the mask position with numpy : 0.0003743171691894531 nb_pixel_total : 8573 time to create 1 rle with old method : 0.011041641235351562 length of segment : 297 time for calcul the mask position with numpy : 0.0006914138793945312 nb_pixel_total : 51341 time to create 1 rle with old method : 0.05822491645812988 length of segment : 192 time for calcul the mask position with numpy : 0.0015273094177246094 nb_pixel_total : 91434 time to create 1 rle with old method : 0.10512208938598633 length of segment : 420 time for calcul the mask position with numpy : 0.0005843639373779297 nb_pixel_total : 13490 time to create 1 rle with old method : 0.015756607055664062 length of segment : 167 time for calcul the mask position with numpy : 0.019138574600219727 nb_pixel_total : 958173 time to create 1 rle with new method : 0.08899712562561035 length of segment : 1537 time for calcul the mask position with numpy : 0.0007855892181396484 nb_pixel_total : 44290 time to create 1 rle with old method : 0.053952693939208984 length of segment : 340 time for calcul the mask position with numpy : 0.0064809322357177734 nb_pixel_total : 311399 time to create 1 rle with new method : 0.011851072311401367 length of segment : 961 time for calcul the mask position with numpy : 0.0023648738861083984 nb_pixel_total : 160242 time to create 1 rle with new method : 0.007630586624145508 length of segment : 557 time for calcul the mask position with numpy : 0.0007517337799072266 nb_pixel_total : 30998 time to create 1 rle with old method : 0.03593301773071289 length of segment : 177 time for calcul the mask position with numpy : 0.000431060791015625 nb_pixel_total : 31179 time to create 1 rle with old method : 0.036025285720825195 length of segment : 139 time for calcul the mask position with numpy : 0.0019488334655761719 nb_pixel_total : 84234 time to create 1 rle with old method : 0.09522199630737305 length of segment : 261 time for calcul the mask position with numpy : 0.0026640892028808594 nb_pixel_total : 104257 time to create 1 rle with old method : 0.11801719665527344 length of segment : 828 time for calcul the mask position with numpy : 0.0024292469024658203 nb_pixel_total : 144327 time to create 1 rle with old method : 0.16042637825012207 length of segment : 848 time for calcul the mask position with numpy : 0.01369619369506836 nb_pixel_total : 787900 time to create 1 rle with new method : 0.06425929069519043 length of segment : 1067 time for calcul the mask position with numpy : 0.00012159347534179688 nb_pixel_total : 1383 time to create 1 rle with old method : 0.001722097396850586 length of segment : 57 time for calcul the mask position with numpy : 0.0007176399230957031 nb_pixel_total : 56612 time to create 1 rle with old method : 0.06485390663146973 length of segment : 215 time for calcul the mask position with numpy : 0.0035681724548339844 nb_pixel_total : 310912 time to create 1 rle with new method : 0.00798654556274414 length of segment : 920 time for calcul the mask position with numpy : 9.870529174804688e-05 nb_pixel_total : 3329 time to create 1 rle with old method : 0.004100799560546875 length of segment : 52 time for calcul the mask position with numpy : 0.00011038780212402344 nb_pixel_total : 1645 time to create 1 rle with old method : 0.0021657943725585938 length of segment : 126 time for calcul the mask position with numpy : 0.002354145050048828 nb_pixel_total : 20168 time to create 1 rle with old method : 0.02438521385192871 length of segment : 578 time for calcul the mask position with numpy : 0.013017892837524414 nb_pixel_total : 461565 time to create 1 rle with new method : 0.020328521728515625 length of segment : 1134 time for calcul the mask position with numpy : 0.0021622180938720703 nb_pixel_total : 88553 time to create 1 rle with old method : 0.09935331344604492 length of segment : 714 time for calcul the mask position with numpy : 0.0013806819915771484 nb_pixel_total : 73391 time to create 1 rle with old method : 0.08337187767028809 length of segment : 399 time for calcul the mask position with numpy : 0.00012230873107910156 nb_pixel_total : 3223 time to create 1 rle with old method : 0.004029750823974609 length of segment : 126 time for calcul the mask position with numpy : 0.0002262592315673828 nb_pixel_total : 9092 time to create 1 rle with old method : 0.010651350021362305 length of segment : 148 time for calcul the mask position with numpy : 0.0018889904022216797 nb_pixel_total : 27226 time to create 1 rle with old method : 0.03216743469238281 length of segment : 495 time for calcul the mask position with numpy : 0.0006215572357177734 nb_pixel_total : 36879 time to create 1 rle with old method : 0.04192376136779785 length of segment : 262 time for calcul the mask position with numpy : 0.00038361549377441406 nb_pixel_total : 14329 time to create 1 rle with old method : 0.01692342758178711 length of segment : 196 time for calcul the mask position with numpy : 0.0006060600280761719 nb_pixel_total : 21941 time to create 1 rle with old method : 0.025498151779174805 length of segment : 386 time for calcul the mask position with numpy : 0.0007779598236083984 nb_pixel_total : 58330 time to create 1 rle with old method : 0.06841540336608887 length of segment : 335 time for calcul the mask position with numpy : 0.0003483295440673828 nb_pixel_total : 13069 time to create 1 rle with old method : 0.015664100646972656 length of segment : 172 time for calcul the mask position with numpy : 0.00034427642822265625 nb_pixel_total : 9107 time to create 1 rle with old method : 0.010783910751342773 length of segment : 140 time for calcul the mask position with numpy : 0.015548467636108398 nb_pixel_total : 1087192 time to create 1 rle with new method : 0.033505916595458984 length of segment : 1012 time for calcul the mask position with numpy : 0.001806020736694336 nb_pixel_total : 125103 time to create 1 rle with old method : 0.13974356651306152 length of segment : 764 time for calcul the mask position with numpy : 0.004438877105712891 nb_pixel_total : 302040 time to create 1 rle with new method : 0.009423017501831055 length of segment : 944 time for calcul the mask position with numpy : 0.0004105567932128906 nb_pixel_total : 23563 time to create 1 rle with old method : 0.027936697006225586 length of segment : 137 time spent for convertir_results : 8.609330415725708 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 76 chid ids of type : 3663 Number RLEs to save : 36203 save missing photos in datou_result : time spend for datou_step_exec : 34.531041860580444 time spend to save output : 2.0742719173431396 total time spend for step 1 : 36.605313777923584 step2:crop_condition Fri Feb 28 08:11:06 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 76 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340085979, 1340085899, 1340085897, 1340085858, 1340085854, 1340085819, 1340085814, 1340085762, 1340085759, 1340085309] Looping around the photos to save general results len do output : 26 /-3689714939Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714941Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714971Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714988Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714990Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689715000Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689715003Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714954Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714983Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714972Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714958Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714974Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714975Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714967Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714993Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689715007Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689715004Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689715008Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689715013Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714964Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714981Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714986Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689715009Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714951Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714970Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689714996Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085979', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085899', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085897', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085858', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085854', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085819', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085814', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085762', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085759', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085309', None, None, None, None, None, '2622650') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 88 time used for this insertion : 0.047667503356933594 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.578515291213989 time spend to save output : 0.04893684387207031 total time spend for step 2 : 7.62745213508606 step3:thcl Fri Feb 28 08:11:14 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.006214618682861328 time to convert the images to numpy array : 1.9073486328125e-06 time to import caffe and check if the image exist : 0.007152080535888672 time to convert the images to numpy array : 0.016926050186157227 time to import caffe and check if the image exist : 0.005750894546508789 time to convert the images to numpy array : 0.022441387176513672 time to import caffe and check if the image exist : 0.013719320297241211 time to convert the images to numpy array : 0.017031431198120117 time to import caffe and check if the image exist : 0.009344100952148438 time to convert the images to numpy array : 0.026380300521850586 time to import caffe and check if the image exist : 0.01031637191772461 time to convert the images to numpy array : 0.026559114456176758 time to import caffe and check if the image exist : 0.011676549911499023 time to convert the images to numpy array : 0.029309988021850586 time to import caffe and check if the image exist : 0.009782791137695312 time to convert the images to numpy array : 0.03603839874267578 time to import caffe and check if the image exist : 0.0069887638092041016 time to convert the images to numpy array : 0.04015207290649414 time to import caffe and check if the image exist : 0.010724067687988281 time to convert the images to numpy array : 0.06919980049133301 total time to convert the images to numpy array : 0.33232998847961426 list photo_ids error: [] list photo_ids correct : [-3689714970, -3689714996, -3689714986, -3689715009, -3689714951, -3689715013, -3689714964, -3689714981, -3689715003, -3689714954, -3689714983, -3689715007, -3689715004, -3689715008, -3689714975, -3689714967, -3689714993, -3689714939, -3689714941, -3689714971, -3689714988, -3689714990, -3689715000, -3689714972, -3689714958, -3689714974] number of photos to traite : 26 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4864 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.27034759521484375 time used to do the prediction : 0.12943601608276367 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4862 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4862 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.33474206924438477 time used to do the prediction : 0.07447481155395508 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 9.059906005859375e-06 save missing photos in datou_result : time spend for datou_step_exec : 7.94159197807312 time spend to save output : 0.00026416778564453125 total time spend for step 3 : 7.941856145858765 step4:merge_mask_thcl_custom Fri Feb 28 08:11:21 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 76 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1340085979, 1340085899, 1340085897, 1340085858, 1340085854, 1340085819, 1340085814, 1340085762, 1340085759, 1340085309] Looping around the photos to save general results len do output : 10 /1340085979Didn't retrieve data .Didn't retrieve data . /1340085899Didn't retrieve data .Didn't retrieve data . /1340085897Didn't retrieve data .Didn't retrieve data . /1340085858Didn't retrieve data .Didn't retrieve data . /1340085854Didn't retrieve data .Didn't retrieve data . /1340085819Didn't retrieve data .Didn't retrieve data . /1340085814Didn't retrieve data .Didn't retrieve data . /1340085762Didn't retrieve data .Didn't retrieve data . /1340085759Didn't retrieve data .Didn't retrieve data . /1340085309Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085979', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085899', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085897', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085858', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085854', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085819', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085814', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085762', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085759', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085309', None, None, None, None, None, '2622650') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.018712520599365234 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.05657649040222168 time spend to save output : 0.019151926040649414 total time spend for step 4 : 0.0757284164428711 step5:rle_unique_nms_with_priority Fri Feb 28 08:11:22 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 10 nb_hashtags : 3 time to prepare the origin masks : 1.7502994537353516 time for calcul the mask position with numpy : 0.05675077438354492 nb_pixel_total : 1493648 time to create 1 rle with new method : 0.3263242244720459 time for calcul the mask position with numpy : 0.011132478713989258 nb_pixel_total : 9189 time to create 1 rle with old method : 0.010686635971069336 time for calcul the mask position with numpy : 0.0072553157806396484 nb_pixel_total : 2810 time to create 1 rle with old method : 0.003456592559814453 time for calcul the mask position with numpy : 0.007930994033813477 nb_pixel_total : 90474 time to create 1 rle with old method : 0.12077808380126953 time for calcul the mask position with numpy : 0.00955343246459961 nb_pixel_total : 288497 time to create 1 rle with new method : 0.28225278854370117 time for calcul the mask position with numpy : 0.007242918014526367 nb_pixel_total : 29101 time to create 1 rle with old method : 0.032987356185913086 time for calcul the mask position with numpy : 0.006803989410400391 nb_pixel_total : 10742 time to create 1 rle with old method : 0.016187429428100586 time for calcul the mask position with numpy : 0.006448268890380859 nb_pixel_total : 6658 time to create 1 rle with old method : 0.008515119552612305 time for calcul the mask position with numpy : 0.006639003753662109 nb_pixel_total : 6042 time to create 1 rle with old method : 0.007177829742431641 time for calcul the mask position with numpy : 0.006298065185546875 nb_pixel_total : 12387 time to create 1 rle with old method : 0.0145416259765625 time for calcul the mask position with numpy : 0.0076334476470947266 nb_pixel_total : 124052 time to create 1 rle with old method : 0.14070725440979004 create new chi : 1.1156978607177734 time to delete rle : 0.01834845542907715 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 7065 TO DO : save crop sub photo not yet done ! save time : 0.3956565856933594 nb_obj : 7 nb_hashtags : 2 time to prepare the origin masks : 1.415043830871582 time for calcul the mask position with numpy : 0.018418312072753906 nb_pixel_total : 967773 time to create 1 rle with new method : 1.2221159934997559 time for calcul the mask position with numpy : 0.008248567581176758 nb_pixel_total : 160673 time to create 1 rle with new method : 0.03188133239746094 time for calcul the mask position with numpy : 0.006753206253051758 nb_pixel_total : 137 time to create 1 rle with old method : 0.00020122528076171875 time for calcul the mask position with numpy : 0.00664830207824707 nb_pixel_total : 11496 time to create 1 rle with old method : 0.012892007827758789 time for calcul the mask position with numpy : 0.008955717086791992 nb_pixel_total : 772 time to create 1 rle with old method : 0.001363992691040039 time for calcul the mask position with numpy : 0.007715702056884766 nb_pixel_total : 3834 time to create 1 rle with old method : 0.00469660758972168 time for calcul the mask position with numpy : 0.007025718688964844 nb_pixel_total : 69941 time to create 1 rle with old method : 0.08251714706420898 time for calcul the mask position with numpy : 0.014269113540649414 nb_pixel_total : 858974 time to create 1 rle with new method : 0.03613448143005371 create new chi : 1.470989465713501 time to delete rle : 0.0009927749633789062 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6381 TO DO : save crop sub photo not yet done ! save time : 0.38007235527038574 nb_obj : 5 nb_hashtags : 2 time to prepare the origin masks : 0.6735067367553711 time for calcul the mask position with numpy : 0.015170097351074219 nb_pixel_total : 1103041 time to create 1 rle with new method : 0.03366208076477051 time for calcul the mask position with numpy : 0.006684303283691406 nb_pixel_total : 45586 time to create 1 rle with old method : 0.051479339599609375 time for calcul the mask position with numpy : 0.01233220100402832 nb_pixel_total : 831139 time to create 1 rle with new method : 0.028936386108398438 time for calcul the mask position with numpy : 0.006366252899169922 nb_pixel_total : 53813 time to create 1 rle with old method : 0.06103658676147461 time for calcul the mask position with numpy : 0.0064029693603515625 nb_pixel_total : 16324 time to create 1 rle with old method : 0.018073558807373047 time for calcul the mask position with numpy : 0.006427288055419922 nb_pixel_total : 23697 time to create 1 rle with old method : 0.029599905014038086 create new chi : 0.27706027030944824 time to delete rle : 0.0005960464477539062 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 5919 TO DO : save crop sub photo not yet done ! save time : 0.3537626266479492 nb_obj : 6 nb_hashtags : 2 time to prepare the origin masks : 0.442462682723999 time for calcul the mask position with numpy : 0.014774322509765625 nb_pixel_total : 1148521 time to create 1 rle with new method : 0.3080265522003174 time for calcul the mask position with numpy : 0.007079601287841797 nb_pixel_total : 7863 time to create 1 rle with old method : 0.0088043212890625 time for calcul the mask position with numpy : 0.007447957992553711 nb_pixel_total : 13646 time to create 1 rle with old method : 0.016072511672973633 time for calcul the mask position with numpy : 0.02517843246459961 nb_pixel_total : 879819 time to create 1 rle with new method : 0.13841581344604492 time for calcul the mask position with numpy : 0.006658315658569336 nb_pixel_total : 5636 time to create 1 rle with old method : 0.006913423538208008 time for calcul the mask position with numpy : 0.006399869918823242 nb_pixel_total : 1525 time to create 1 rle with old method : 0.0017268657684326172 time for calcul the mask position with numpy : 0.006858110427856445 nb_pixel_total : 16590 time to create 1 rle with old method : 0.018431663513183594 create new chi : 0.5816650390625 time to delete rle : 0.0004799365997314453 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 4730 TO DO : save crop sub photo not yet done ! save time : 0.3009531497955322 nb_obj : 9 nb_hashtags : 5 time to prepare the origin masks : 1.3131928443908691 time for calcul the mask position with numpy : 0.06167912483215332 nb_pixel_total : 1415813 time to create 1 rle with new method : 0.24624180793762207 time for calcul the mask position with numpy : 0.007164716720581055 nb_pixel_total : 50019 time to create 1 rle with old method : 0.061482906341552734 time for calcul the mask position with numpy : 0.006917476654052734 nb_pixel_total : 1869 time to create 1 rle with old method : 0.0023889541625976562 time for calcul the mask position with numpy : 0.008079290390014648 nb_pixel_total : 7911 time to create 1 rle with old method : 0.008942842483520508 time for calcul the mask position with numpy : 0.006666421890258789 nb_pixel_total : 9431 time to create 1 rle with old method : 0.01069951057434082 time for calcul the mask position with numpy : 0.007773399353027344 nb_pixel_total : 77419 time to create 1 rle with old method : 0.0906212329864502 time for calcul the mask position with numpy : 0.011088371276855469 nb_pixel_total : 355913 time to create 1 rle with new method : 0.0877685546875 time for calcul the mask position with numpy : 0.007134675979614258 nb_pixel_total : 13297 time to create 1 rle with old method : 0.016085147857666016 time for calcul the mask position with numpy : 0.015218019485473633 nb_pixel_total : 125063 time to create 1 rle with old method : 0.17109942436218262 time for calcul the mask position with numpy : 0.006987094879150391 nb_pixel_total : 16865 time to create 1 rle with old method : 0.023537635803222656 create new chi : 0.8683929443359375 time to delete rle : 0.0011334419250488281 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 7159 TO DO : save crop sub photo not yet done ! save time : 0.4375617504119873 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.8630890846252441 time for calcul the mask position with numpy : 0.02936267852783203 nb_pixel_total : 1026930 time to create 1 rle with new method : 0.05920720100402832 time for calcul the mask position with numpy : 0.006758213043212891 nb_pixel_total : 12723 time to create 1 rle with old method : 0.014725208282470703 time for calcul the mask position with numpy : 0.0066356658935546875 nb_pixel_total : 19126 time to create 1 rle with old method : 0.027738094329833984 time for calcul the mask position with numpy : 0.015099287033081055 nb_pixel_total : 756671 time to create 1 rle with new method : 0.03799581527709961 time for calcul the mask position with numpy : 0.009194612503051758 nb_pixel_total : 258150 time to create 1 rle with new method : 0.03281450271606445 create new chi : 0.24862265586853027 time to delete rle : 0.0007750988006591797 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 6681 TO DO : save crop sub photo not yet done ! save time : 0.39885783195495605 nb_obj : 6 nb_hashtags : 3 time to prepare the origin masks : 0.6099302768707275 time for calcul the mask position with numpy : 0.012836217880249023 nb_pixel_total : 918316 time to create 1 rle with new method : 0.056195974349975586 time for calcul the mask position with numpy : 0.0073130130767822266 nb_pixel_total : 52508 time to create 1 rle with old method : 0.06129765510559082 time for calcul the mask position with numpy : 0.006694793701171875 nb_pixel_total : 63707 time to create 1 rle with old method : 0.07115793228149414 time for calcul the mask position with numpy : 0.006808280944824219 nb_pixel_total : 42034 time to create 1 rle with old method : 0.046547889709472656 time for calcul the mask position with numpy : 0.006235837936401367 nb_pixel_total : 26146 time to create 1 rle with old method : 0.02922654151916504 time for calcul the mask position with numpy : 0.006522178649902344 nb_pixel_total : 13447 time to create 1 rle with old method : 0.01498103141784668 time for calcul the mask position with numpy : 0.012561321258544922 nb_pixel_total : 957442 time to create 1 rle with new method : 0.02827620506286621 create new chi : 0.367673397064209 time to delete rle : 0.0005812644958496094 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 6796 TO DO : save crop sub photo not yet done ! save time : 0.40196967124938965 nb_obj : 9 nb_hashtags : 2 time to prepare the origin masks : 1.2079482078552246 time for calcul the mask position with numpy : 0.038993120193481445 nb_pixel_total : 760295 time to create 1 rle with new method : 0.1783456802368164 time for calcul the mask position with numpy : 0.006924867630004883 nb_pixel_total : 1834 time to create 1 rle with old method : 0.0022330284118652344 time for calcul the mask position with numpy : 0.006239175796508789 nb_pixel_total : 906 time to create 1 rle with old method : 0.001074075698852539 time for calcul the mask position with numpy : 0.007776737213134766 nb_pixel_total : 136115 time to create 1 rle with old method : 0.15743160247802734 time for calcul the mask position with numpy : 0.006205081939697266 nb_pixel_total : 1237 time to create 1 rle with old method : 0.001432657241821289 time for calcul the mask position with numpy : 0.006752729415893555 nb_pixel_total : 55281 time to create 1 rle with old method : 0.06447410583496094 time for calcul the mask position with numpy : 0.015383243560791016 nb_pixel_total : 785795 time to create 1 rle with new method : 0.11259579658508301 time for calcul the mask position with numpy : 0.012077093124389648 nb_pixel_total : 104278 time to create 1 rle with old method : 0.11854982376098633 time for calcul the mask position with numpy : 0.007338762283325195 nb_pixel_total : 143880 time to create 1 rle with old method : 0.1673583984375 time for calcul the mask position with numpy : 0.007770538330078125 nb_pixel_total : 83979 time to create 1 rle with old method : 0.12142157554626465 create new chi : 1.0628585815429688 time to delete rle : 0.0010056495666503906 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 8775 TO DO : save crop sub photo not yet done ! save time : 0.5142133235931396 nb_obj : 11 nb_hashtags : 6 time to prepare the origin masks : 1.7748732566833496 time for calcul the mask position with numpy : 0.11940217018127441 nb_pixel_total : 1320655 time to create 1 rle with new method : 0.08845162391662598 time for calcul the mask position with numpy : 0.006021976470947266 nb_pixel_total : 12942 time to create 1 rle with old method : 0.015216827392578125 time for calcul the mask position with numpy : 0.010178089141845703 nb_pixel_total : 36802 time to create 1 rle with old method : 0.03960585594177246 time for calcul the mask position with numpy : 0.009575605392456055 nb_pixel_total : 58175 time to create 1 rle with old method : 0.0607759952545166 time for calcul the mask position with numpy : 0.006410360336303711 nb_pixel_total : 73984 time to create 1 rle with old method : 0.08054256439208984 time for calcul the mask position with numpy : 0.006178617477416992 nb_pixel_total : 22001 time to create 1 rle with old method : 0.025264739990234375 time for calcul the mask position with numpy : 0.006328105926513672 nb_pixel_total : 14302 time to create 1 rle with old method : 0.01614665985107422 time for calcul the mask position with numpy : 0.007976293563842773 nb_pixel_total : 4618 time to create 1 rle with old method : 0.005283832550048828 time for calcul the mask position with numpy : 0.00649571418762207 nb_pixel_total : 451 time to create 1 rle with old method : 0.0005650520324707031 time for calcul the mask position with numpy : 0.006550312042236328 nb_pixel_total : 42175 time to create 1 rle with old method : 0.047948598861694336 time for calcul the mask position with numpy : 0.008980512619018555 nb_pixel_total : 460417 time to create 1 rle with new method : 0.25493359565734863 time for calcul the mask position with numpy : 0.006665945053100586 nb_pixel_total : 27078 time to create 1 rle with old method : 0.030586719512939453 create new chi : 0.8765830993652344 time to delete rle : 0.0011858940124511719 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 8921 TO DO : save crop sub photo not yet done ! save time : 0.5104937553405762 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 0.4589829444885254 time for calcul the mask position with numpy : 0.012100696563720703 nb_pixel_total : 798218 time to create 1 rle with new method : 0.03108692169189453 time for calcul the mask position with numpy : 0.006270170211791992 nb_pixel_total : 9090 time to create 1 rle with old method : 0.009791851043701172 time for calcul the mask position with numpy : 0.00603938102722168 nb_pixel_total : 22131 time to create 1 rle with old method : 0.02428913116455078 time for calcul the mask position with numpy : 0.006245613098144531 nb_pixel_total : 31941 time to create 1 rle with old method : 0.03542280197143555 time for calcul the mask position with numpy : 0.00689387321472168 nb_pixel_total : 125053 time to create 1 rle with old method : 0.16388535499572754 time for calcul the mask position with numpy : 0.013846158981323242 nb_pixel_total : 1087167 time to create 1 rle with new method : 0.03356599807739258 create new chi : 0.3503124713897705 time to delete rle : 0.0005488395690917969 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 5835 TO DO : save crop sub photo not yet done ! save time : 0.361600399017334 map_output_result : {1340085979: (0.0, 'Should be the crop_list due to order', 0), 1340085899: (0.0, 'Should be the crop_list due to order', 0), 1340085897: (0.0, 'Should be the crop_list due to order', 0), 1340085858: (0.0, 'Should be the crop_list due to order', 0), 1340085854: (0.0, 'Should be the crop_list due to order', 0), 1340085819: (0.0, 'Should be the crop_list due to order', 0), 1340085814: (0.0, 'Should be the crop_list due to order', 0), 1340085762: (0.0, 'Should be the crop_list due to order', 0), 1340085759: (0.0, 'Should be the crop_list due to order', 0), 1340085309: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1340085979, 1340085899, 1340085897, 1340085858, 1340085854, 1340085819, 1340085814, 1340085762, 1340085759, 1340085309] Looping around the photos to save general results len do output : 10 /1340085979.Didn't retrieve data . /1340085899.Didn't retrieve data . /1340085897.Didn't retrieve data . /1340085858.Didn't retrieve data . /1340085854.Didn't retrieve data . /1340085819.Didn't retrieve data . /1340085814.Didn't retrieve data . /1340085762.Didn't retrieve data . /1340085759.Didn't retrieve data . /1340085309.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085979', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085899', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085897', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085858', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085854', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085819', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085814', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085762', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085759', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085309', None, None, None, None, None, '2622650') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01306009292602539 save_final save missing photos in datou_result : time spend for datou_step_exec : 22.157617330551147 time spend to save output : 0.013550519943237305 total time spend for step 5 : 22.171167850494385 step6:crop_condition Fri Feb 28 08:11:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 82 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740726705_3785188 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7639033794403076 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740726706_3785188 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6338851451873779 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740726707_3785188 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.4362659454345703 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740726708_3785188 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.584291934967041 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740726709_3785188 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.3225674629211426 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740726711_3785188 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5358083248138428 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340085979, 1340085899, 1340085897, 1340085858, 1340085854, 1340085819, 1340085814, 1340085762, 1340085759, 1340085309] Looping around the photos to save general results len do output : 10 /1340092016Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340092017Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340092019Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340092020Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340092021Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340092024Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340092027Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340092028Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340092029Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340092031Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085979', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085899', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085897', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085858', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085854', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085819', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085814', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085762', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085759', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085309', None, None, None, None, None, '2622650') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 40 time used for this insertion : 0.022507190704345703 save_final save missing photos in datou_result : time spend for datou_step_exec : 8.149682998657227 time spend to save output : 0.023190736770629883 total time spend for step 6 : 8.172873735427856 step7:ventilate_hashtags_in_portfolio Fri Feb 28 08:11:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20915779 get user id for portfolio 20915779 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20915779 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('plastique','mal_croppe','cartonnette','Carton_brun','autre_refus','metal','Teint_Dans_La_Masse','flou','papier','Carton_gris','environnement','kraft')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20915779 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('plastique','mal_croppe','cartonnette','Carton_brun','autre_refus','metal','Teint_Dans_La_Masse','flou','papier','Carton_gris','environnement','kraft')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20915779 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('plastique','mal_croppe','cartonnette','Carton_brun','autre_refus','metal','Teint_Dans_La_Masse','flou','papier','Carton_gris','environnement','kraft')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20916051,20916052,20916053,20916054,20916055,20916056,20916057,20916058,20916059,20916060,20916061,20916062?tags=plastique,mal_croppe,cartonnette,Carton_brun,autre_refus,metal,Teint_Dans_La_Masse,flou,papier,Carton_gris,environnement,kraft Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1340085979, 1340085899, 1340085897, 1340085858, 1340085854, 1340085819, 1340085814, 1340085762, 1340085759, 1340085309] Looping around the photos to save general results len do output : 1 /20915779. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085979', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085899', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085897', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085858', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085854', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085819', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085814', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085762', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085759', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085309', None, None, None, None, None, '2622650') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01365041732788086 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.6777312755584717 time spend to save output : 0.013924360275268555 total time spend for step 7 : 0.6916556358337402 step8:final Fri Feb 28 08:11:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1340085979: ('0.04017470717428347',), 1340085899: ('0.04017470717428347',), 1340085897: ('0.04017470717428347',), 1340085858: ('0.04017470717428347',), 1340085854: ('0.04017470717428347',), 1340085819: ('0.04017470717428347',), 1340085814: ('0.04017470717428347',), 1340085762: ('0.04017470717428347',), 1340085759: ('0.04017470717428347',), 1340085309: ('0.04017470717428347',)} new output for save of step final : {1340085979: ('0.04017470717428347',), 1340085899: ('0.04017470717428347',), 1340085897: ('0.04017470717428347',), 1340085858: ('0.04017470717428347',), 1340085854: ('0.04017470717428347',), 1340085819: ('0.04017470717428347',), 1340085814: ('0.04017470717428347',), 1340085762: ('0.04017470717428347',), 1340085759: ('0.04017470717428347',), 1340085309: ('0.04017470717428347',)} [1340085979, 1340085899, 1340085897, 1340085858, 1340085854, 1340085819, 1340085814, 1340085762, 1340085759, 1340085309] Looping around the photos to save general results len do output : 10 /1340085979.Didn't retrieve data . /1340085899.Didn't retrieve data . /1340085897.Didn't retrieve data . /1340085858.Didn't retrieve data . /1340085854.Didn't retrieve data . /1340085819.Didn't retrieve data . /1340085814.Didn't retrieve data . /1340085762.Didn't retrieve data . /1340085759.Didn't retrieve data . /1340085309.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085979', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085899', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085897', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085858', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085854', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085819', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085814', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085762', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085759', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085309', None, None, None, None, None, '2622650') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.012712717056274414 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.09678769111633301 time spend to save output : 0.013113260269165039 total time spend for step 8 : 0.10990095138549805 step9:velours_tree Fri Feb 28 08:11:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.035817623138427734 time spend to save output : 3.5762786865234375e-05 total time spend for step 9 : 0.03585338592529297 step10:send_mail_cod Fri Feb 28 08:11:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20915779_28-02-2025_08_11_53.pdf 20916051 change filename to text .imagette209160511740726713 20916052 imagette209160521740726713 20916053 change filename to text .imagette209160531740726713 20916055 change filename to text .change filename to text .imagette209160551740726713 20916056 imagette209160561740726713 20916058 imagette209160581740726713 20916062 imagette209160621740726713 20916054 change filename to text .change filename to text .change filename to text .imagette209160541740726713 20916057 change filename to text .change filename to text .imagette209160571740726713 20916060 change filename to text .imagette209160601740726713 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20915779 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20916051,20916052,20916053,20916054,20916055,20916056,20916057,20916058,20916059,20916060,20916061,20916062?tags=plastique,mal_croppe,cartonnette,Carton_brun,autre_refus,metal,Teint_Dans_La_Masse,flou,papier,Carton_gris,environnement,kraft your option no_mail is active, we will not send the real mail to your client args[1340085979] : ('0.04017470717428347',) no score found for photo 1340085979 We are sending mail with results at report@fotonower.com args[1340085899] : ('0.04017470717428347',) no score found for photo 1340085899 We are sending mail with results at report@fotonower.com args[1340085897] : ('0.04017470717428347',) no score found for photo 1340085897 We are sending mail with results at report@fotonower.com args[1340085858] : ('0.04017470717428347',) no score found for photo 1340085858 We are sending mail with results at report@fotonower.com args[1340085854] : ('0.04017470717428347',) no score found for photo 1340085854 We are sending mail with results at report@fotonower.com args[1340085819] : ('0.04017470717428347',) no score found for photo 1340085819 We are sending mail with results at report@fotonower.com args[1340085814] : ('0.04017470717428347',) no score found for photo 1340085814 We are sending mail with results at report@fotonower.com args[1340085762] : ('0.04017470717428347',) no score found for photo 1340085762 We are sending mail with results at report@fotonower.com args[1340085759] : ('0.04017470717428347',) no score found for photo 1340085759 We are sending mail with results at report@fotonower.com args[1340085309] : ('0.04017470717428347',) no score found for photo 1340085309 We are sending mail with results at report@fotonower.com refus_total : 0.04017470717428347 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20915779 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1340085858,1340085759,1340085139,1340085854,1340085200,1340085979,1340085899,1340085897,1340085819,1340085814,1340085762,1340085309,1340083994,1340085198,1340085143,1340085059,1340085052,1340084952,1340084663,1340084122) Found this number of photos: 20 begin to download photo : 1340085858 begin to download photo : 1340085979 begin to download photo : 1340085762 begin to download photo : 1340085059 download finish for photo 1340085762 begin to download photo : 1340085309 download finish for photo 1340085858 begin to download photo : 1340085759 download finish for photo 1340085059 begin to download photo : 1340085052 download finish for photo 1340085979 begin to download photo : 1340085899 download finish for photo 1340085309 begin to download photo : 1340083994 download finish for photo 1340085052 begin to download photo : 1340084952 download finish for photo 1340085759 begin to download photo : 1340085139 download finish for photo 1340085899 begin to download photo : 1340085897 download finish for photo 1340083994 begin to download photo : 1340085198 download finish for photo 1340084952 begin to download photo : 1340084663 download finish for photo 1340085897 begin to download photo : 1340085819 download finish for photo 1340085139 begin to download photo : 1340085854 download finish for photo 1340085198 begin to download photo : 1340085143 download finish for photo 1340084663 begin to download photo : 1340084122 download finish for photo 1340085854 begin to download photo : 1340085200 download finish for photo 1340085819 begin to download photo : 1340085814 download finish for photo 1340085143 download finish for photo 1340084122 download finish for photo 1340085200 download finish for photo 1340085814 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20915779_28-02-2025_08_11_53.pdf results_Auto_P20915779_28-02-2025_08_11_53.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20915779_28-02-2025_08_11_53.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20915779','results_Auto_P20915779_28-02-2025_08_11_53.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20915779_28-02-2025_08_11_53.pdf','pdf','','0.21','0.04017470717428347') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1340085979, 1340085899, 1340085897, 1340085858, 1340085854, 1340085819, 1340085814, 1340085762, 1340085759, 1340085309] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085979', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085899', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085897', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085858', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085854', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085819', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085814', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085762', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085759', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085309', None, None, None, None, None, '2622650') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.012732982635498047 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.7364437580108643 time spend to save output : 0.012898921966552734 total time spend for step 10 : 2.749342679977417 step11:split_time_score Fri Feb 28 08:11:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('07', 22),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 28022025 20915779 Nombre de photos uploadées : 22 / 23040 (0%) 28022025 20915779 Nombre de photos taguées (types de déchets): 0 / 22 (0%) 28022025 20915779 Nombre de photos taguées (volume) : 0 / 22 (0%) elapsed_time : load_data_split_time_score 1.6689300537109375e-06 elapsed_time : order_list_meta_photo_and_scores 4.76837158203125e-06 ?????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.001033782958984375 elapsed_time : insert_dashboard_record_day_entry 0.022916078567504883 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.05088624831321179 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20914260_28-02-2025_07_41_57.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20914260 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20914260 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20915778 order by id desc limit 1 Qualite : 0.04017470717428347 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20915779_28-02-2025_08_11_53.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20915779 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20915779 AND mptpi.`type`=3726 To do Qualite : 0.03410941429420713 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20915780_28-02-2025_08_09_46.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20915780 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20915780 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'28022025': {'nb_upload': 22, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1340085979, 1340085899, 1340085897, 1340085858, 1340085854, 1340085819, 1340085814, 1340085762, 1340085759, 1340085309] Looping around the photos to save general results len do output : 1 /20915779Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085979', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085899', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085897', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085858', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085854', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085819', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085814', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085762', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085759', None, None, None, None, None, '2622650') ('3459', None, None, None, None, None, None, None, '2622650') ('3459', '20915779', '1340085309', None, None, None, None, None, '2622650') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013524770736694336 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.1466848850250244 time spend to save output : 0.013668537139892578 total time spend for step 11 : 1.160353422164917 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 49.26user 32.39system 1:31.94elapsed 88%CPU (0avgtext+0avgdata 2797404maxresident)k 466592inputs+28960outputs (832major+2318050minor)pagefaults 0swaps