python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2227327 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 29>10 please execute split_portfolio.py -i 20597799 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2592394'] with mtr_portfolio_ids : ['20597799'] and first list_photo_ids : [] new path : /proc/2227327/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 0.944643497467041 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Mon Feb 17 08:02:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-17 08:02:32.157553: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-17 08:02:32.183144: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-17 08:02:32.185199: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f9790000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-17 08:02:32.185229: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-17 08:02:32.189304: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-17 08:02:32.451391: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x42baeef0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-17 08:02:32.451437: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-17 08:02:32.452685: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-17 08:02:32.453044: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-17 08:02:32.459927: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-17 08:02:32.462680: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-17 08:02:32.463160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-17 08:02:32.466708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-17 08:02:32.468222: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-17 08:02:32.472597: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-17 08:02:32.474075: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-17 08:02:32.474136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-17 08:02:32.474931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-17 08:02:32.474948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-17 08:02:32.474957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-17 08:02:32.476374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-17 08:02:32.751692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-17 08:02:32.751805: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-17 08:02:32.751833: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-17 08:02:32.751859: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-17 08:02:32.751883: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-17 08:02:32.751908: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-17 08:02:32.751932: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-17 08:02:32.751957: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-17 08:02:32.753696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-17 08:02:32.756044: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-17 08:02:32.756196: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-17 08:02:32.756222: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-17 08:02:32.756246: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-17 08:02:32.756269: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-17 08:02:32.756291: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-17 08:02:32.756312: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-17 08:02:32.756335: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-17 08:02:32.757940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-17 08:02:32.757980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-17 08:02:32.757992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-17 08:02:32.758001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-17 08:02:32.759669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-17 08:02:40.992518: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-17 08:02:41.180503: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 1 Detection mask done ! Trying to reset tf kernel 2227832 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5485 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10774 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.037293434143066406 nb_pixel_total : 956994 time to create 1 rle with new method : 0.08123993873596191 length of segment : 2019 time for calcul the mask position with numpy : 0.004079341888427734 nb_pixel_total : 298881 time to create 1 rle with new method : 0.015186071395874023 length of segment : 1284 time for calcul the mask position with numpy : 0.0041561126708984375 nb_pixel_total : 309616 time to create 1 rle with new method : 0.00975346565246582 length of segment : 1172 time for calcul the mask position with numpy : 0.010741710662841797 nb_pixel_total : 632914 time to create 1 rle with new method : 0.05433464050292969 length of segment : 1102 time for calcul the mask position with numpy : 0.002321004867553711 nb_pixel_total : 173893 time to create 1 rle with new method : 0.005183696746826172 length of segment : 748 time for calcul the mask position with numpy : 0.001504659652709961 nb_pixel_total : 85690 time to create 1 rle with old method : 0.09659194946289062 length of segment : 536 time for calcul the mask position with numpy : 0.0003285408020019531 nb_pixel_total : 16089 time to create 1 rle with old method : 0.01750040054321289 length of segment : 117 time for calcul the mask position with numpy : 0.01801156997680664 nb_pixel_total : 746929 time to create 1 rle with new method : 0.06885027885437012 length of segment : 1461 time for calcul the mask position with numpy : 0.0004146099090576172 nb_pixel_total : 17028 time to create 1 rle with old method : 0.019610166549682617 length of segment : 294 time for calcul the mask position with numpy : 0.0013701915740966797 nb_pixel_total : 102043 time to create 1 rle with old method : 0.10455083847045898 length of segment : 290 time for calcul the mask position with numpy : 0.00016117095947265625 nb_pixel_total : 6129 time to create 1 rle with old method : 0.006880998611450195 length of segment : 105 time for calcul the mask position with numpy : 0.0007030963897705078 nb_pixel_total : 49881 time to create 1 rle with old method : 0.054215192794799805 length of segment : 289 time for calcul the mask position with numpy : 0.00019359588623046875 nb_pixel_total : 8346 time to create 1 rle with old method : 0.00968027114868164 length of segment : 117 time for calcul the mask position with numpy : 0.0052337646484375 nb_pixel_total : 295742 time to create 1 rle with new method : 0.010415792465209961 length of segment : 997 time for calcul the mask position with numpy : 0.007733345031738281 nb_pixel_total : 452519 time to create 1 rle with new method : 0.12130618095397949 length of segment : 1650 time for calcul the mask position with numpy : 0.004235982894897461 nb_pixel_total : 308542 time to create 1 rle with new method : 0.007704257965087891 length of segment : 993 time for calcul the mask position with numpy : 0.01665973663330078 nb_pixel_total : 1305401 time to create 1 rle with new method : 0.06788754463195801 length of segment : 2034 time for calcul the mask position with numpy : 0.0002772808074951172 nb_pixel_total : 20557 time to create 1 rle with old method : 0.023312807083129883 length of segment : 125 time for calcul the mask position with numpy : 0.0019605159759521484 nb_pixel_total : 150703 time to create 1 rle with new method : 0.00445866584777832 length of segment : 656 time for calcul the mask position with numpy : 0.003304004669189453 nb_pixel_total : 300588 time to create 1 rle with new method : 0.00701594352722168 length of segment : 925 time for calcul the mask position with numpy : 0.0038824081420898438 nb_pixel_total : 162359 time to create 1 rle with new method : 0.007848262786865234 length of segment : 1201 time for calcul the mask position with numpy : 0.0011768341064453125 nb_pixel_total : 88391 time to create 1 rle with old method : 0.09300971031188965 length of segment : 404 time for calcul the mask position with numpy : 0.011890172958374023 nb_pixel_total : 794453 time to create 1 rle with new method : 0.16936302185058594 length of segment : 1258 time for calcul the mask position with numpy : 0.0028264522552490234 nb_pixel_total : 143520 time to create 1 rle with old method : 0.16055846214294434 length of segment : 783 time for calcul the mask position with numpy : 0.01787734031677246 nb_pixel_total : 888951 time to create 1 rle with new method : 0.1253490447998047 length of segment : 1294 time for calcul the mask position with numpy : 0.010367631912231445 nb_pixel_total : 623199 time to create 1 rle with new method : 0.04304957389831543 length of segment : 1087 time for calcul the mask position with numpy : 0.002134561538696289 nb_pixel_total : 150771 time to create 1 rle with new method : 0.005567312240600586 length of segment : 618 time for calcul the mask position with numpy : 0.0011534690856933594 nb_pixel_total : 89114 time to create 1 rle with old method : 0.10107660293579102 length of segment : 532 time for calcul the mask position with numpy : 0.06821393966674805 nb_pixel_total : 976470 time to create 1 rle with new method : 0.054691314697265625 length of segment : 1696 time for calcul the mask position with numpy : 0.0035932064056396484 nb_pixel_total : 284692 time to create 1 rle with new method : 0.00910043716430664 length of segment : 1111 time for calcul the mask position with numpy : 0.003636598587036133 nb_pixel_total : 305639 time to create 1 rle with new method : 0.007642507553100586 length of segment : 947 time for calcul the mask position with numpy : 0.0828397274017334 nb_pixel_total : 1746352 time to create 1 rle with new method : 0.06076622009277344 length of segment : 1197 time spent for convertir_results : 6.723215579986572 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 32 chid ids of type : 3663 Number RLEs to save : 29042 save missing photos in datou_result : time spend for datou_step_exec : 28.267279863357544 time spend to save output : 1.6073908805847168 total time spend for step 1 : 29.87467074394226 step2:crop_condition Mon Feb 17 08:02:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 32 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1338119101, 1338119089, 1338119041, 1338119038, 1338119008, 1338119005, 1338118964, 1338118960, 1338118826, 1338118823] Looping around the photos to save general results len do output : 6 /-3677726159Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677726157Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677726160Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677726165Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677726175Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3677726158Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119101', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119089', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119041', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119038', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119008', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119005', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118964', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118960', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118826', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118823', None, None, None, None, None, '2592394') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 28 time used for this insertion : 0.019587039947509766 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.1140220165252686 time spend to save output : 0.020101547241210938 total time spend for step 2 : 2.1341235637664795 step3:thcl Mon Feb 17 08:03:01 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.005509138107299805 time to convert the images to numpy array : 0.002408266067504883 time to import caffe and check if the image exist : 0.0051648616790771484 time to convert the images to numpy array : 0.0024764537811279297 time to import caffe and check if the image exist : 0.004670143127441406 time to convert the images to numpy array : 0.003984928131103516 time to import caffe and check if the image exist : 0.006438493728637695 time to convert the images to numpy array : 0.0041735172271728516 time to import caffe and check if the image exist : 0.005978822708129883 time to convert the images to numpy array : 0.009679079055786133 time to import caffe and check if the image exist : 0.005232572555541992 time to convert the images to numpy array : 0.009942054748535156 total time to convert the images to numpy array : 0.2295234203338623 list photo_ids error: [] list photo_ids correct : [-3677726160, -3677726158, -3677726165, -3677726159, -3677726157, -3677726175] number of photos to traite : 6 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10555 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.05538487434387207 time used to do the prediction : 0.07157516479492188 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10553 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10553 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.08189868927001953 time used to do the prediction : 0.040802955627441406 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.0013580322265625e-05 save missing photos in datou_result : time spend for datou_step_exec : 6.943062782287598 time spend to save output : 0.00015783309936523438 total time spend for step 3 : 6.943220615386963 step4:merge_mask_thcl_custom Mon Feb 17 08:03:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 32 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1338119101, 1338119089, 1338119041, 1338119038, 1338119008, 1338119005, 1338118964, 1338118960, 1338118826, 1338118823] Looping around the photos to save general results len do output : 10 /1338119101Didn't retrieve data .Didn't retrieve data . /1338119089Didn't retrieve data .Didn't retrieve data . /1338119041Didn't retrieve data .Didn't retrieve data . /1338119038Didn't retrieve data .Didn't retrieve data . /1338119008Didn't retrieve data .Didn't retrieve data . /1338119005Didn't retrieve data .Didn't retrieve data . /1338118964Didn't retrieve data .Didn't retrieve data . /1338118960Didn't retrieve data .Didn't retrieve data . /1338118826Didn't retrieve data .Didn't retrieve data . /1338118823Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119101', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119089', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119041', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119038', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119008', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119005', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118964', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118960', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118826', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118823', None, None, None, None, None, '2592394') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.0142822265625 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.05262589454650879 time spend to save output : 0.014791727066040039 total time spend for step 4 : 0.06741762161254883 step5:rle_unique_nms_with_priority Mon Feb 17 08:03:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.9698131084442139 time for calcul the mask position with numpy : 0.03347420692443848 nb_pixel_total : 969053 time to create 1 rle with new method : 0.07457566261291504 time for calcul the mask position with numpy : 0.0071489810943603516 nb_pixel_total : 133845 time to create 1 rle with old method : 0.14057064056396484 time for calcul the mask position with numpy : 0.006298065185546875 nb_pixel_total : 16147 time to create 1 rle with old method : 0.017547130584716797 time for calcul the mask position with numpy : 0.19603180885314941 nb_pixel_total : 954555 time to create 1 rle with new method : 0.09509396553039551 create new chi : 0.5872552394866943 time to delete rle : 0.026438474655151367 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 7150 TO DO : save crop sub photo not yet done ! save time : 0.4096853733062744 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 0.4713327884674072 time for calcul the mask position with numpy : 0.038387298583984375 nb_pixel_total : 1267417 time to create 1 rle with new method : 0.1253213882446289 time for calcul the mask position with numpy : 0.008119583129882812 nb_pixel_total : 173763 time to create 1 rle with new method : 0.07602047920227051 time for calcul the mask position with numpy : 0.014499664306640625 nb_pixel_total : 632420 time to create 1 rle with new method : 0.03238725662231445 create new chi : 0.30396103858947754 time to delete rle : 0.0007479190826416016 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 4796 TO DO : save crop sub photo not yet done ! save time : 0.2879679203033447 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 1.544083833694458 time for calcul the mask position with numpy : 0.05019950866699219 nb_pixel_total : 979126 time to create 1 rle with new method : 0.1014564037322998 time for calcul the mask position with numpy : 0.007744789123535156 nb_pixel_total : 101382 time to create 1 rle with old method : 0.1409132480621338 time for calcul the mask position with numpy : 0.00903940200805664 nb_pixel_total : 5898 time to create 1 rle with old method : 0.010219097137451172 time for calcul the mask position with numpy : 0.008920907974243164 nb_pixel_total : 100062 time to create 1 rle with old method : 0.11816883087158203 time for calcul the mask position with numpy : 0.007188081741333008 nb_pixel_total : 36140 time to create 1 rle with old method : 0.050980329513549805 time for calcul the mask position with numpy : 0.010219097137451172 nb_pixel_total : 8262 time to create 1 rle with old method : 0.014199495315551758 time for calcul the mask position with numpy : 0.009647846221923828 nb_pixel_total : 11582 time to create 1 rle with old method : 0.019428491592407227 time for calcul the mask position with numpy : 0.022758960723876953 nb_pixel_total : 729595 time to create 1 rle with new method : 0.2511250972747803 time for calcul the mask position with numpy : 0.006768941879272461 nb_pixel_total : 15978 time to create 1 rle with old method : 0.01741623878479004 time for calcul the mask position with numpy : 0.007121086120605469 nb_pixel_total : 85575 time to create 1 rle with old method : 0.1379408836364746 create new chi : 1.0183184146881104 time to delete rle : 0.0006685256958007812 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 7779 TO DO : save crop sub photo not yet done ! save time : 0.4570462703704834 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 0.14965581893920898 time for calcul the mask position with numpy : 0.014992952346801758 nb_pixel_total : 1316352 time to create 1 rle with new method : 0.03320908546447754 time for calcul the mask position with numpy : 0.007959127426147461 nb_pixel_total : 308042 time to create 1 rle with new method : 0.02951979637145996 time for calcul the mask position with numpy : 0.009067773818969727 nb_pixel_total : 449206 time to create 1 rle with new method : 0.03057122230529785 create new chi : 0.1341557502746582 time to delete rle : 0.0006074905395507812 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 6358 TO DO : save crop sub photo not yet done ! save time : 0.37167811393737793 nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.43064355850219727 time for calcul the mask position with numpy : 0.011246442794799805 nb_pixel_total : 540135 time to create 1 rle with new method : 0.21665453910827637 time for calcul the mask position with numpy : 0.006368160247802734 nb_pixel_total : 20487 time to create 1 rle with old method : 0.02186727523803711 time for calcul the mask position with numpy : 0.006129026412963867 nb_pixel_total : 41307 time to create 1 rle with old method : 0.04403090476989746 time for calcul the mask position with numpy : 0.006546735763549805 nb_pixel_total : 49682 time to create 1 rle with old method : 0.051619768142700195 time for calcul the mask position with numpy : 0.2696413993835449 nb_pixel_total : 1421989 time to create 1 rle with new method : 0.07539534568786621 create new chi : 0.7251214981079102 time to delete rle : 0.0005965232849121094 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 5930 TO DO : save crop sub photo not yet done ! save time : 0.34705138206481934 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.41504907608032227 time for calcul the mask position with numpy : 0.019042491912841797 nb_pixel_total : 1093425 time to create 1 rle with new method : 0.23736095428466797 time for calcul the mask position with numpy : 0.013706684112548828 nb_pixel_total : 798288 time to create 1 rle with new method : 0.05405163764953613 time for calcul the mask position with numpy : 0.006119966506958008 nb_pixel_total : 20137 time to create 1 rle with old method : 0.021454811096191406 time for calcul the mask position with numpy : 0.006593227386474609 nb_pixel_total : 161750 time to create 1 rle with new method : 0.02818751335144043 create new chi : 0.3942286968231201 time to delete rle : 0.0005140304565429688 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 5913 TO DO : save crop sub photo not yet done ! save time : 0.3472940921783447 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 0.5129866600036621 time for calcul the mask position with numpy : 0.036966562271118164 nb_pixel_total : 1060126 time to create 1 rle with new method : 0.20214438438415527 time for calcul the mask position with numpy : 0.012033939361572266 nb_pixel_total : 870300 time to create 1 rle with new method : 0.0813453197479248 time for calcul the mask position with numpy : 0.007143497467041016 nb_pixel_total : 143174 time to create 1 rle with old method : 0.14473962783813477 create new chi : 0.49976515769958496 time to delete rle : 0.0004413127899169922 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 4895 TO DO : save crop sub photo not yet done ! save time : 0.3204779624938965 nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 0.5882713794708252 time for calcul the mask position with numpy : 0.015433073043823242 nb_pixel_total : 1214587 time to create 1 rle with new method : 0.23779726028442383 time for calcul the mask position with numpy : 0.0070040225982666016 nb_pixel_total : 140733 time to create 1 rle with old method : 0.14318180084228516 time for calcul the mask position with numpy : 0.010843753814697266 nb_pixel_total : 622422 time to create 1 rle with new method : 0.07206559181213379 time for calcul the mask position with numpy : 0.006383657455444336 nb_pixel_total : 95858 time to create 1 rle with old method : 0.09902000427246094 create new chi : 0.6008424758911133 time to delete rle : 0.0005514621734619141 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 5092 TO DO : save crop sub photo not yet done ! save time : 0.37691521644592285 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.44281625747680664 time for calcul the mask position with numpy : 0.013198614120483398 nb_pixel_total : 932791 time to create 1 rle with new method : 0.12560272216796875 time for calcul the mask position with numpy : 0.007021903991699219 nb_pixel_total : 87842 time to create 1 rle with old method : 0.09995341300964355 time for calcul the mask position with numpy : 0.006609678268432617 nb_pixel_total : 74524 time to create 1 rle with old method : 0.0823049545288086 time for calcul the mask position with numpy : 0.012773513793945312 nb_pixel_total : 978443 time to create 1 rle with new method : 0.19159483909606934 create new chi : 0.557750940322876 time to delete rle : 0.0005462169647216797 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 6045 TO DO : save crop sub photo not yet done ! save time : 0.3485555648803711 nb_obj : 1 nb_hashtags : 1 time to prepare the origin masks : 0.11736011505126953 time for calcul the mask position with numpy : 0.007800102233886719 nb_pixel_total : 328650 time to create 1 rle with new method : 0.07628822326660156 time for calcul the mask position with numpy : 0.017473459243774414 nb_pixel_total : 1744950 time to create 1 rle with new method : 0.040801048278808594 create new chi : 0.1493527889251709 time to delete rle : 0.00038504600524902344 batch 1 Loaded 2 chid ids of type : 3726 Number RLEs to save : 3464 TO DO : save crop sub photo not yet done ! save time : 0.2088615894317627 map_output_result : {1338119101: (0.0, 'Should be the crop_list due to order', 0.0), 1338119089: (0.0, 'Should be the crop_list due to order', 0.0), 1338119041: (0.0, 'Should be the crop_list due to order', 0.0), 1338119038: (0.0, 'Should be the crop_list due to order', 0.0), 1338119008: (0.0, 'Should be the crop_list due to order', 0.0), 1338119005: (0.0, 'Should be the crop_list due to order', 0.0), 1338118964: (0.0, 'Should be the crop_list due to order', 0.0), 1338118960: (0.0, 'Should be the crop_list due to order', 0.0), 1338118826: (0.0, 'Should be the crop_list due to order', 0.0), 1338118823: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1338119101, 1338119089, 1338119041, 1338119038, 1338119008, 1338119005, 1338118964, 1338118960, 1338118826, 1338118823] Looping around the photos to save general results len do output : 10 /1338119101.Didn't retrieve data . /1338119089.Didn't retrieve data . /1338119041.Didn't retrieve data . /1338119038.Didn't retrieve data . /1338119008.Didn't retrieve data . /1338119005.Didn't retrieve data . /1338118964.Didn't retrieve data . /1338118960.Didn't retrieve data . /1338118826.Didn't retrieve data . /1338118823.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119101', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119089', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119041', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119038', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119008', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119005', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118964', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118960', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118826', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118823', None, None, None, None, None, '2592394') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01284646987915039 save_final save missing photos in datou_result : time spend for datou_step_exec : 14.405101776123047 time spend to save output : 0.013224363327026367 total time spend for step 5 : 14.418326139450073 step6:crop_condition Mon Feb 17 08:03:22 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 42 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739775803_2227327 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6621448993682861 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739775804_2227327 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6097090244293213 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739775805_2227327 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5484235286712646 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1338119101, 1338119089, 1338119041, 1338119038, 1338119008, 1338119005, 1338118964, 1338118960, 1338118826, 1338118823] Looping around the photos to save general results len do output : 4 /1338119285Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338119286Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338119287Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338119289Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119101', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119089', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119041', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119038', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119008', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119005', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118964', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118960', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118826', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118823', None, None, None, None, None, '2592394') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 22 time used for this insertion : 0.020487070083618164 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.5169098377227783 time spend to save output : 0.020916223526000977 total time spend for step 6 : 3.5378260612487793 step7:ventilate_hashtags_in_portfolio Mon Feb 17 08:03:26 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20597799 get user id for portfolio 20597799 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20597799 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','flou','metal','Teint_Dans_La_Masse','mal_croppe','cartonnette','kraft','Carton_brun','environnement','Carton_gris','plastique','autre_refus')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20597799 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','flou','metal','Teint_Dans_La_Masse','mal_croppe','cartonnette','kraft','Carton_brun','environnement','Carton_gris','plastique','autre_refus')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20597799 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','flou','metal','Teint_Dans_La_Masse','mal_croppe','cartonnette','kraft','Carton_brun','environnement','Carton_gris','plastique','autre_refus')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20597806,20597807,20597808,20597809,20597810,20597811,20597812,20597813,20597814,20597815,20597816,20597817?tags=papier,flou,metal,Teint_Dans_La_Masse,mal_croppe,cartonnette,kraft,Carton_brun,environnement,Carton_gris,plastique,autre_refus Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1338119101, 1338119089, 1338119041, 1338119038, 1338119008, 1338119005, 1338118964, 1338118960, 1338118826, 1338118823] Looping around the photos to save general results len do output : 1 /20597799. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119101', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119089', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119041', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119038', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119008', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119005', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118964', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118960', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118826', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118823', None, None, None, None, None, '2592394') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014551639556884766 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.64479398727417 time spend to save output : 0.014849424362182617 total time spend for step 7 : 1.6596434116363525 step8:final Mon Feb 17 08:03:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1338119101: ('0.022240673507391933',), 1338119089: ('0.022240673507391933',), 1338119041: ('0.022240673507391933',), 1338119038: ('0.022240673507391933',), 1338119008: ('0.022240673507391933',), 1338119005: ('0.022240673507391933',), 1338118964: ('0.022240673507391933',), 1338118960: ('0.022240673507391933',), 1338118826: ('0.022240673507391933',), 1338118823: ('0.022240673507391933',)} new output for save of step final : {1338119101: ('0.022240673507391933',), 1338119089: ('0.022240673507391933',), 1338119041: ('0.022240673507391933',), 1338119038: ('0.022240673507391933',), 1338119008: ('0.022240673507391933',), 1338119005: ('0.022240673507391933',), 1338118964: ('0.022240673507391933',), 1338118960: ('0.022240673507391933',), 1338118826: ('0.022240673507391933',), 1338118823: ('0.022240673507391933',)} [1338119101, 1338119089, 1338119041, 1338119038, 1338119008, 1338119005, 1338118964, 1338118960, 1338118826, 1338118823] Looping around the photos to save general results len do output : 10 /1338119101.Didn't retrieve data . /1338119089.Didn't retrieve data . /1338119041.Didn't retrieve data . /1338119038.Didn't retrieve data . /1338119008.Didn't retrieve data . /1338119005.Didn't retrieve data . /1338118964.Didn't retrieve data . /1338118960.Didn't retrieve data . /1338118826.Didn't retrieve data . /1338118823.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119101', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119089', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119041', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119038', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119008', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119005', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118964', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118960', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118826', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118823', None, None, None, None, None, '2592394') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.0131378173828125 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.0970604419708252 time spend to save output : 0.013666629791259766 total time spend for step 8 : 0.11072707176208496 step9:velours_tree Mon Feb 17 08:03:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.03000926971435547 time spend to save output : 3.647804260253906e-05 total time spend for step 9 : 0.030045747756958008 step10:send_mail_cod Mon Feb 17 08:03:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20597799_17-02-2025_08_03_28.pdf 20597807 imagette205978071739775808 20597808 imagette205978081739775808 20597810 imagette205978101739775808 20597811 change filename to text .imagette205978111739775808 20597812 imagette205978121739775808 20597816 change filename to text .imagette205978161739775808 20597817 imagette205978171739775808 20597809 change filename to text .change filename to text .imagette205978091739775808 20597813 imagette205978131739775808 20597815 imagette205978151739775808 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20597799 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20597806,20597807,20597808,20597809,20597810,20597811,20597812,20597813,20597814,20597815,20597816,20597817?tags=papier,flou,metal,Teint_Dans_La_Masse,mal_croppe,cartonnette,kraft,Carton_brun,environnement,Carton_gris,plastique,autre_refus your option no_mail is active, we will not send the real mail to your client args[1338119101] : ('0.022240673507391933',) no score found for photo 1338119101 apple ('0.022240673507391933',) We are sending mail with results at report@fotonower.com args[1338119089] : ('0.022240673507391933',) no score found for photo 1338119089 apple ('0.022240673507391933',) We are sending mail with results at report@fotonower.com args[1338119041] : ('0.022240673507391933',) no score found for photo 1338119041 apple ('0.022240673507391933',) We are sending mail with results at report@fotonower.com args[1338119038] : ('0.022240673507391933',) no score found for photo 1338119038 apple ('0.022240673507391933',) We are sending mail with results at report@fotonower.com args[1338119008] : ('0.022240673507391933',) no score found for photo 1338119008 apple ('0.022240673507391933',) We are sending mail with results at report@fotonower.com args[1338119005] : ('0.022240673507391933',) no score found for photo 1338119005 apple ('0.022240673507391933',) We are sending mail with results at report@fotonower.com args[1338118964] : ('0.022240673507391933',) no score found for photo 1338118964 apple ('0.022240673507391933',) We are sending mail with results at report@fotonower.com args[1338118960] : ('0.022240673507391933',) no score found for photo 1338118960 apple ('0.022240673507391933',) We are sending mail with results at report@fotonower.com args[1338118826] : ('0.022240673507391933',) no score found for photo 1338118826 apple ('0.022240673507391933',) We are sending mail with results at report@fotonower.com args[1338118823] : ('0.022240673507391933',) no score found for photo 1338118823 apple ('0.022240673507391933',) We are sending mail with results at report@fotonower.com refus_total : 0.022240673507391933 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20597799 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1338118960,1338118789,1338118693,1338119101,1338119089,1338119041,1338119038,1338119008,1338119005,1338118964,1338118826,1338118823,1338118781,1338118733,1338118730,1338118244,1338118691,1338118648,1338118645,1338118605) Found this number of photos: 20 begin to download photo : 1338118960 begin to download photo : 1338119041 begin to download photo : 1338118826 begin to download photo : 1338118244 download finish for photo 1338118826 begin to download photo : 1338118823 download finish for photo 1338119041 begin to download photo : 1338119038 download finish for photo 1338118960 begin to download photo : 1338118789 download finish for photo 1338118244 begin to download photo : 1338118691 download finish for photo 1338118789 begin to download photo : 1338118693 download finish for photo 1338119038 begin to download photo : 1338119008 download finish for photo 1338118823 begin to download photo : 1338118781 download finish for photo 1338118691 begin to download photo : 1338118648 download finish for photo 1338119008 begin to download photo : 1338119005 download finish for photo 1338118693 begin to download photo : 1338119101 download finish for photo 1338118781 begin to download photo : 1338118733 download finish for photo 1338118648 begin to download photo : 1338118645 download finish for photo 1338119005 begin to download photo : 1338118964 download finish for photo 1338119101 begin to download photo : 1338119089 download finish for photo 1338118733 begin to download photo : 1338118730 download finish for photo 1338118645 begin to download photo : 1338118605 download finish for photo 1338118964 download finish for photo 1338119089 download finish for photo 1338118730 download finish for photo 1338118605 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20597799_17-02-2025_08_03_28.pdf results_Auto_P20597799_17-02-2025_08_03_28.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20597799_17-02-2025_08_03_28.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20597799','results_Auto_P20597799_17-02-2025_08_03_28.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20597799_17-02-2025_08_03_28.pdf','pdf','','0.17','0.022240673507391933') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1338119101, 1338119089, 1338119041, 1338119038, 1338119008, 1338119005, 1338118964, 1338118960, 1338118826, 1338118823] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119101', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119089', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119041', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119038', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119008', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119005', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118964', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118960', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118826', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118823', None, None, None, None, None, '2592394') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.014741182327270508 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.443248987197876 time spend to save output : 0.015120506286621094 total time spend for step 10 : 2.458369493484497 step11:split_time_score Mon Feb 17 08:03:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('07', 29),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 17022025 20597799 Nombre de photos uploadées : 29 / 23040 (0%) 17022025 20597799 Nombre de photos taguées (types de déchets): 0 / 29 (0%) 17022025 20597799 Nombre de photos taguées (volume) : 0 / 29 (0%) elapsed_time : load_data_split_time_score 3.5762786865234375e-06 elapsed_time : order_list_meta_photo_and_scores 9.298324584960938e-06 ????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0015304088592529297 elapsed_time : insert_dashboard_record_day_entry 0.02226114273071289 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20597796 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20597797 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20597798 order by id desc limit 1 Qualite : 0.022240673507391933 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20597799_17-02-2025_08_03_28.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20597799 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20597799 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'17022025': {'nb_upload': 29, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1338119101, 1338119089, 1338119041, 1338119038, 1338119008, 1338119005, 1338118964, 1338118960, 1338118826, 1338118823] Looping around the photos to save general results len do output : 1 /20597799Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119101', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119089', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119041', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119038', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119008', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338119005', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118964', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118960', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118826', None, None, None, None, None, '2592394') ('3459', None, None, None, None, None, None, None, '2592394') ('3459', '20597799', '1338118823', None, None, None, None, None, '2592394') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013529539108276367 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.506362199783325 time spend to save output : 0.0137786865234375 total time spend for step 11 : 2.5201408863067627 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 31.99user 25.58system 1:08.53elapsed 84%CPU (0avgtext+0avgdata 2738148maxresident)k 555616inputs+19528outputs (202major+2025697minor)pagefaults 0swaps