python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3917509 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 22>10 please execute split_portfolio.py -i 21050245 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2636165'] with mtr_portfolio_ids : ['21050245'] and first list_photo_ids : [] new path : /proc/3917509/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.177506685256958 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Wed Mar 5 07:38:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10372 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-05 07:38:31.167749: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-05 07:38:31.176367: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-05 07:38:31.177864: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f40bc000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-05 07:38:31.177920: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-05 07:38:31.180937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-05 07:38:31.422680: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1fa73610 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-05 07:38:31.422740: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-05 07:38:31.424279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 07:38:31.424755: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 07:38:31.429335: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 07:38:31.432619: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 07:38:31.432950: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 07:38:31.436567: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 07:38:31.437651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 07:38:31.442656: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 07:38:31.444231: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 07:38:31.444317: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 07:38:31.445062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 07:38:31.445078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 07:38:31.445087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 07:38:31.446354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9607 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-05 07:38:31.721042: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 07:38:31.721142: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 07:38:31.721164: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 07:38:31.721184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 07:38:31.721202: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 07:38:31.721220: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 07:38:31.721239: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 07:38:31.721257: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 07:38:31.722787: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 07:38:31.724113: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 07:38:31.724153: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 07:38:31.724175: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 07:38:31.724196: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 07:38:31.724217: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 07:38:31.724237: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 07:38:31.724263: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 07:38:31.724284: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 07:38:31.725586: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 07:38:31.725612: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 07:38:31.725620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 07:38:31.725628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 07:38:31.726878: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9607 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-05 07:38:39.383118: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 07:38:39.558409: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 1 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 Detection mask done ! Trying to reset tf kernel 3918090 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5083 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10372 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0005705356597900391 nb_pixel_total : 22471 time to create 1 rle with old method : 0.023963212966918945 length of segment : 246 time for calcul the mask position with numpy : 0.028333663940429688 nb_pixel_total : 765926 time to create 1 rle with new method : 0.16536355018615723 length of segment : 1703 time for calcul the mask position with numpy : 0.0005674362182617188 nb_pixel_total : 29007 time to create 1 rle with old method : 0.031725406646728516 length of segment : 476 time for calcul the mask position with numpy : 0.0032205581665039062 nb_pixel_total : 168754 time to create 1 rle with new method : 0.010998010635375977 length of segment : 697 time for calcul the mask position with numpy : 0.0038764476776123047 nb_pixel_total : 288704 time to create 1 rle with new method : 0.009348869323730469 length of segment : 1238 time for calcul the mask position with numpy : 0.0007996559143066406 nb_pixel_total : 45944 time to create 1 rle with old method : 0.05092930793762207 length of segment : 329 time for calcul the mask position with numpy : 0.018003225326538086 nb_pixel_total : 950288 time to create 1 rle with new method : 0.0855247974395752 length of segment : 1214 time for calcul the mask position with numpy : 0.0020575523376464844 nb_pixel_total : 79117 time to create 1 rle with old method : 0.09158492088317871 length of segment : 516 time for calcul the mask position with numpy : 0.0020334720611572266 nb_pixel_total : 93949 time to create 1 rle with old method : 0.12357139587402344 length of segment : 500 time for calcul the mask position with numpy : 0.2541842460632324 nb_pixel_total : 1828854 time to create 1 rle with new method : 0.08991503715515137 length of segment : 1083 time for calcul the mask position with numpy : 0.0067064762115478516 nb_pixel_total : 317537 time to create 1 rle with new method : 0.008970975875854492 length of segment : 958 time for calcul the mask position with numpy : 0.028494596481323242 nb_pixel_total : 1739377 time to create 1 rle with new method : 0.08111739158630371 length of segment : 1468 time for calcul the mask position with numpy : 0.021413087844848633 nb_pixel_total : 1430400 time to create 1 rle with new method : 0.07690906524658203 length of segment : 1488 time for calcul the mask position with numpy : 0.003286600112915039 nb_pixel_total : 224322 time to create 1 rle with new method : 0.009134054183959961 length of segment : 1058 time for calcul the mask position with numpy : 0.003433704376220703 nb_pixel_total : 318736 time to create 1 rle with new method : 0.007748126983642578 length of segment : 926 time for calcul the mask position with numpy : 0.05854988098144531 nb_pixel_total : 1807267 time to create 1 rle with new method : 0.07102108001708984 length of segment : 1318 time for calcul the mask position with numpy : 0.005492210388183594 nb_pixel_total : 313869 time to create 1 rle with new method : 0.01104283332824707 length of segment : 981 time for calcul the mask position with numpy : 0.0002505779266357422 nb_pixel_total : 6431 time to create 1 rle with old method : 0.007699012756347656 length of segment : 69 time for calcul the mask position with numpy : 0.002431631088256836 nb_pixel_total : 94900 time to create 1 rle with old method : 0.10291433334350586 length of segment : 580 time for calcul the mask position with numpy : 0.010758399963378906 nb_pixel_total : 458398 time to create 1 rle with new method : 0.0630497932434082 length of segment : 1040 time for calcul the mask position with numpy : 0.0011446475982666016 nb_pixel_total : 63242 time to create 1 rle with old method : 0.06993985176086426 length of segment : 341 time for calcul the mask position with numpy : 0.0036802291870117188 nb_pixel_total : 315642 time to create 1 rle with new method : 0.008075952529907227 length of segment : 1060 time for calcul the mask position with numpy : 0.0004601478576660156 nb_pixel_total : 19399 time to create 1 rle with old method : 0.0223848819732666 length of segment : 173 time for calcul the mask position with numpy : 0.0016522407531738281 nb_pixel_total : 110460 time to create 1 rle with old method : 0.12289118766784668 length of segment : 421 time for calcul the mask position with numpy : 0.001657247543334961 nb_pixel_total : 54337 time to create 1 rle with old method : 0.06142115592956543 length of segment : 400 time for calcul the mask position with numpy : 0.0017070770263671875 nb_pixel_total : 111506 time to create 1 rle with old method : 0.12455415725708008 length of segment : 773 time for calcul the mask position with numpy : 0.004475593566894531 nb_pixel_total : 185526 time to create 1 rle with new method : 0.00718379020690918 length of segment : 669 time for calcul the mask position with numpy : 0.02367854118347168 nb_pixel_total : 1363192 time to create 1 rle with new method : 0.08170700073242188 length of segment : 1950 time for calcul the mask position with numpy : 0.00017404556274414062 nb_pixel_total : 7682 time to create 1 rle with old method : 0.008948087692260742 length of segment : 90 time for calcul the mask position with numpy : 0.0038208961486816406 nb_pixel_total : 302880 time to create 1 rle with new method : 0.008387088775634766 length of segment : 958 time for calcul the mask position with numpy : 0.004162311553955078 nb_pixel_total : 158301 time to create 1 rle with new method : 0.008419990539550781 length of segment : 932 time for calcul the mask position with numpy : 0.015372514724731445 nb_pixel_total : 923081 time to create 1 rle with new method : 0.19699573516845703 length of segment : 1159 time for calcul the mask position with numpy : 0.0043048858642578125 nb_pixel_total : 309920 time to create 1 rle with new method : 0.010655641555786133 length of segment : 1098 time for calcul the mask position with numpy : 0.021905183792114258 nb_pixel_total : 1090717 time to create 1 rle with new method : 0.07481002807617188 length of segment : 2104 time for calcul the mask position with numpy : 0.0035669803619384766 nb_pixel_total : 295813 time to create 1 rle with new method : 0.008363723754882812 length of segment : 854 time for calcul the mask position with numpy : 0.0010676383972167969 nb_pixel_total : 41638 time to create 1 rle with old method : 0.04741621017456055 length of segment : 560 time spent for convertir_results : 7.4680211544036865 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 36 chid ids of type : 3663 Number RLEs to save : 31430 save missing photos in datou_result : time spend for datou_step_exec : 28.313541889190674 time spend to save output : 13.158008098602295 total time spend for step 1 : 41.47154998779297 step2:crop_condition Wed Mar 5 07:39:09 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 36 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340952880, 1340952860, 1340952781, 1340952777, 1340952734, 1340952715, 1340952584, 1340952423, 1340952292, 1340952290] Looping around the photos to save general results len do output : 7 /-3694149602Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694149617Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694149590Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694149589Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694149607Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694149605Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3694149599Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952880', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952860', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952781', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952777', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952734', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952715', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952584', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952423', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952292', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952290', None, None, None, None, None, '2636165') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 31 time used for this insertion : 0.01869654655456543 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.5033159255981445 time spend to save output : 0.019420623779296875 total time spend for step 2 : 3.5227365493774414 step3:thcl Wed Mar 5 07:39:13 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.006746053695678711 time to convert the images to numpy array : 0.0019137859344482422 time to import caffe and check if the image exist : 0.0037407875061035156 time to convert the images to numpy array : 0.011165142059326172 time to import caffe and check if the image exist : 0.006623268127441406 time to convert the images to numpy array : 0.011361122131347656 time to import caffe and check if the image exist : 0.003935575485229492 time to convert the images to numpy array : 0.015220165252685547 time to import caffe and check if the image exist : 0.0052068233489990234 time to convert the images to numpy array : 0.017755508422851562 time to import caffe and check if the image exist : 0.004409313201904297 time to convert the images to numpy array : 0.018962383270263672 time to import caffe and check if the image exist : 0.0053195953369140625 time to convert the images to numpy array : 0.020986080169677734 total time to convert the images to numpy array : 0.25374388694763184 list photo_ids error: [] list photo_ids correct : [-3694149599, -3694149617, -3694149602, -3694149590, -3694149605, -3694149589, -3694149607] number of photos to traite : 7 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10372 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10153 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.09462761878967285 time used to do the prediction : 0.0684354305267334 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10151 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10151 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.15282440185546875 time used to do the prediction : 0.04093647003173828 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.9311904907226562e-05 save missing photos in datou_result : time spend for datou_step_exec : 7.377005100250244 time spend to save output : 0.0002353191375732422 total time spend for step 3 : 7.377240419387817 step4:merge_mask_thcl_custom Wed Mar 5 07:39:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 36 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1340952880, 1340952860, 1340952781, 1340952777, 1340952734, 1340952715, 1340952584, 1340952423, 1340952292, 1340952290] Looping around the photos to save general results len do output : 10 /1340952880Didn't retrieve data .Didn't retrieve data . /1340952860Didn't retrieve data .Didn't retrieve data . /1340952781Didn't retrieve data .Didn't retrieve data . /1340952777Didn't retrieve data .Didn't retrieve data . /1340952734Didn't retrieve data .Didn't retrieve data . /1340952715Didn't retrieve data .Didn't retrieve data . /1340952584Didn't retrieve data .Didn't retrieve data . /1340952423Didn't retrieve data .Didn't retrieve data . /1340952292Didn't retrieve data .Didn't retrieve data . /1340952290Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952880', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952860', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952781', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952777', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952734', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952715', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952584', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952423', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952292', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952290', None, None, None, None, None, '2636165') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.012337207794189453 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.049898386001586914 time spend to save output : 0.012879371643066406 total time spend for step 4 : 0.06277775764465332 step5:rle_unique_nms_with_priority Wed Mar 5 07:39:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 5 nb_hashtags : 1 time to prepare the origin masks : 0.8052010536193848 time for calcul the mask position with numpy : 0.03997659683227539 nb_pixel_total : 1082664 time to create 1 rle with new method : 0.2653517723083496 time for calcul the mask position with numpy : 0.0067327022552490234 nb_pixel_total : 43701 time to create 1 rle with old method : 0.050194740295410156 time for calcul the mask position with numpy : 0.0066950321197509766 nb_pixel_total : 41267 time to create 1 rle with old method : 0.04581713676452637 time for calcul the mask position with numpy : 0.006690263748168945 nb_pixel_total : 29048 time to create 1 rle with old method : 0.03246021270751953 time for calcul the mask position with numpy : 0.013701200485229492 nb_pixel_total : 854454 time to create 1 rle with new method : 0.16289210319519043 time for calcul the mask position with numpy : 0.007002592086791992 nb_pixel_total : 22466 time to create 1 rle with old method : 0.036356210708618164 create new chi : 0.691411018371582 time to delete rle : 0.02598118782043457 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6165 TO DO : save crop sub photo not yet done ! save time : 1.6311304569244385 nb_obj : 4 nb_hashtags : 3 time to prepare the origin masks : 0.5391654968261719 time for calcul the mask position with numpy : 0.013303756713867188 nb_pixel_total : 963625 time to create 1 rle with new method : 0.08383059501647949 time for calcul the mask position with numpy : 0.006942033767700195 nb_pixel_total : 68086 time to create 1 rle with old method : 0.07898902893066406 time for calcul the mask position with numpy : 0.006787538528442383 nb_pixel_total : 47812 time to create 1 rle with old method : 0.059488773345947266 time for calcul the mask position with numpy : 0.014025449752807617 nb_pixel_total : 948237 time to create 1 rle with new method : 0.1133885383605957 time for calcul the mask position with numpy : 0.008585214614868164 nb_pixel_total : 45840 time to create 1 rle with old method : 0.06085491180419922 create new chi : 0.45332813262939453 time to delete rle : 0.0010075569152832031 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 5475 TO DO : save crop sub photo not yet done ! save time : 2.9254329204559326 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 0.27220940589904785 time for calcul the mask position with numpy : 0.008590221405029297 nb_pixel_total : 199517 time to create 1 rle with new method : 0.04818582534790039 time for calcul the mask position with numpy : 0.006793975830078125 nb_pixel_total : 44917 time to create 1 rle with old method : 0.05049014091491699 time for calcul the mask position with numpy : 0.026492834091186523 nb_pixel_total : 1829166 time to create 1 rle with new method : 0.07447504997253418 create new chi : 0.2220444679260254 time to delete rle : 0.0003681182861328125 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 3984 TO DO : save crop sub photo not yet done ! save time : 3.978654384613037 nb_obj : 1 nb_hashtags : 1 time to prepare the origin masks : 0.20647048950195312 time for calcul the mask position with numpy : 0.009101390838623047 nb_pixel_total : 322838 time to create 1 rle with new method : 0.1805417537689209 time for calcul the mask position with numpy : 0.061810970306396484 nb_pixel_total : 1750762 time to create 1 rle with new method : 0.11063432693481445 create new chi : 0.37880921363830566 time to delete rle : 0.0003857612609863281 batch 1 Loaded 2 chid ids of type : 3726 Number RLEs to save : 3742 TO DO : save crop sub photo not yet done ! save time : 0.2431795597076416 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.5194211006164551 time for calcul the mask position with numpy : 0.010299444198608398 nb_pixel_total : 454395 time to create 1 rle with new method : 0.08420300483703613 time for calcul the mask position with numpy : 0.007687091827392578 nb_pixel_total : 158371 time to create 1 rle with new method : 0.09945535659790039 time for calcul the mask position with numpy : 0.007398843765258789 nb_pixel_total : 34061 time to create 1 rle with old method : 0.03819990158081055 time for calcul the mask position with numpy : 0.01694798469543457 nb_pixel_total : 1426773 time to create 1 rle with new method : 0.054711341857910156 create new chi : 0.33324217796325684 time to delete rle : 0.0006151199340820312 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 6372 TO DO : save crop sub photo not yet done ! save time : 1.3231801986694336 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 0.3577144145965576 time for calcul the mask position with numpy : 0.008779287338256836 nb_pixel_total : 200558 time to create 1 rle with new method : 0.16706609725952148 time for calcul the mask position with numpy : 0.0072689056396484375 nb_pixel_total : 48077 time to create 1 rle with old method : 0.06353092193603516 time for calcul the mask position with numpy : 0.022768020629882812 nb_pixel_total : 1824965 time to create 1 rle with new method : 0.06406664848327637 create new chi : 0.34079408645629883 time to delete rle : 0.0005452632904052734 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 3997 TO DO : save crop sub photo not yet done ! save time : 2.379887104034424 nb_obj : 8 nb_hashtags : 4 time to prepare the origin masks : 0.9592018127441406 time for calcul the mask position with numpy : 0.01806807518005371 nb_pixel_total : 968977 time to create 1 rle with new method : 0.20827722549438477 time for calcul the mask position with numpy : 0.007462024688720703 nb_pixel_total : 62696 time to create 1 rle with old method : 0.06815099716186523 time for calcul the mask position with numpy : 0.006651639938354492 nb_pixel_total : 19347 time to create 1 rle with old method : 0.021600961685180664 time for calcul the mask position with numpy : 0.007619619369506836 nb_pixel_total : 231232 time to create 1 rle with new method : 0.08473777770996094 time for calcul the mask position with numpy : 0.008510589599609375 nb_pixel_total : 114054 time to create 1 rle with old method : 0.1260688304901123 time for calcul the mask position with numpy : 0.010137557983398438 nb_pixel_total : 465564 time to create 1 rle with new method : 0.028587818145751953 time for calcul the mask position with numpy : 0.006696224212646484 nb_pixel_total : 110545 time to create 1 rle with old method : 0.12443757057189941 time for calcul the mask position with numpy : 0.0066242218017578125 nb_pixel_total : 6360 time to create 1 rle with old method : 0.010223150253295898 time for calcul the mask position with numpy : 0.007487297058105469 nb_pixel_total : 94825 time to create 1 rle with old method : 0.11671662330627441 create new chi : 0.8764894008636475 time to delete rle : 0.0008132457733154297 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 8638 TO DO : save crop sub photo not yet done ! save time : 2.2191879749298096 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.40721607208251953 time for calcul the mask position with numpy : 0.01072549819946289 nb_pixel_total : 587080 time to create 1 rle with new method : 0.04836583137512207 time for calcul the mask position with numpy : 0.006587505340576172 nb_pixel_total : 43584 time to create 1 rle with old method : 0.07554459571838379 time for calcul the mask position with numpy : 0.02403545379638672 nb_pixel_total : 7593 time to create 1 rle with old method : 0.012369155883789062 time for calcul the mask position with numpy : 0.02175617218017578 nb_pixel_total : 1250036 time to create 1 rle with new method : 0.03849220275878906 time for calcul the mask position with numpy : 0.011448860168457031 nb_pixel_total : 185307 time to create 1 rle with new method : 0.02963733673095703 create new chi : 0.2864372730255127 time to delete rle : 0.0005321502685546875 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 5810 TO DO : save crop sub photo not yet done ! save time : 3.6404342651367188 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.2936227321624756 time for calcul the mask position with numpy : 0.014212608337402344 nb_pixel_total : 895594 time to create 1 rle with new method : 0.04004621505737305 time for calcul the mask position with numpy : 0.00647425651550293 nb_pixel_total : 90309 time to create 1 rle with old method : 0.09629297256469727 time for calcul the mask position with numpy : 0.012778043746948242 nb_pixel_total : 929778 time to create 1 rle with new method : 0.028431415557861328 time for calcul the mask position with numpy : 0.006983518600463867 nb_pixel_total : 157919 time to create 1 rle with new method : 0.0284268856048584 create new chi : 0.23446893692016602 time to delete rle : 0.0006241798400878906 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 5886 TO DO : save crop sub photo not yet done ! save time : 2.8792996406555176 nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 0.3545348644256592 time for calcul the mask position with numpy : 0.012659549713134766 nb_pixel_total : 813058 time to create 1 rle with new method : 0.03457808494567871 time for calcul the mask position with numpy : 0.006433725357055664 nb_pixel_total : 35342 time to create 1 rle with old method : 0.03803372383117676 time for calcul the mask position with numpy : 0.00686192512512207 nb_pixel_total : 129454 time to create 1 rle with old method : 0.16709232330322266 time for calcul the mask position with numpy : 0.014638185501098633 nb_pixel_total : 1095746 time to create 1 rle with new method : 0.029248476028442383 create new chi : 0.3104512691497803 time to delete rle : 0.0006549358367919922 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 7041 TO DO : save crop sub photo not yet done ! save time : 3.150998115539551 map_output_result : {1340952880: (0.0, 'Should be the crop_list due to order', 0), 1340952860: (0.0, 'Should be the crop_list due to order', 0), 1340952781: (0.0, 'Should be the crop_list due to order', 0), 1340952777: (0.0, 'Should be the crop_list due to order', 0), 1340952734: (0.0, 'Should be the crop_list due to order', 0), 1340952715: (0.0, 'Should be the crop_list due to order', 0), 1340952584: (0.0, 'Should be the crop_list due to order', 0), 1340952423: (0.0, 'Should be the crop_list due to order', 0), 1340952292: (0.0, 'Should be the crop_list due to order', 0), 1340952290: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1340952880, 1340952860, 1340952781, 1340952777, 1340952734, 1340952715, 1340952584, 1340952423, 1340952292, 1340952290] Looping around the photos to save general results len do output : 10 /1340952880.Didn't retrieve data . /1340952860.Didn't retrieve data . /1340952781.Didn't retrieve data . /1340952777.Didn't retrieve data . /1340952734.Didn't retrieve data . /1340952715.Didn't retrieve data . /1340952584.Didn't retrieve data . /1340952423.Didn't retrieve data . /1340952292.Didn't retrieve data . /1340952290.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952880', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952860', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952781', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952777', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952734', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952715', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952584', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952423', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952292', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952290', None, None, None, None, None, '2636165') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014501094818115234 save_final save missing photos in datou_result : time spend for datou_step_exec : 33.579047441482544 time spend to save output : 0.01501154899597168 total time spend for step 5 : 33.594058990478516 step6:crop_condition Wed Mar 5 07:39:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 45 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741156795_3917509 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5500304698944092 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741156796_3917509 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6825506687164307 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340952880, 1340952860, 1340952781, 1340952777, 1340952734, 1340952715, 1340952584, 1340952423, 1340952292, 1340952290] Looping around the photos to save general results len do output : 2 /1340956819Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340956820Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952880', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952860', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952781', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952777', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952734', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952715', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952584', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952423', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952292', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952290', None, None, None, None, None, '2636165') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 16 time used for this insertion : 0.6739592552185059 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.0024900436401367 time spend to save output : 0.6743416786193848 total time spend for step 6 : 3.6768317222595215 step7:ventilate_hashtags_in_portfolio Wed Mar 5 07:39:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 21050245 get user id for portfolio 21050245 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21050245 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','metal','mal_croppe','autre_refus','plastique','Teint_Dans_La_Masse','flou','Carton_gris','cartonnette','Carton_brun','kraft','environnement')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21050245 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','metal','mal_croppe','autre_refus','plastique','Teint_Dans_La_Masse','flou','Carton_gris','cartonnette','Carton_brun','kraft','environnement')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/21050320,21050321,21050322,21050323,21050324,21050325,21050326,21050327,21050328,21050329,21050330,21050331?tags=metal,Carton_gris,kraft,Carton_brun,autre_refus,papier,plastique,mal_croppe,flou,environnement,cartonnette,Teint_Dans_La_Masse Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1340952880, 1340952860, 1340952781, 1340952777, 1340952734, 1340952715, 1340952584, 1340952423, 1340952292, 1340952290] Looping around the photos to save general results len do output : 1 /21050245. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952880', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952860', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952781', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952777', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952734', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952715', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952584', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952423', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952292', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952290', None, None, None, None, None, '2636165') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01485586166381836 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.4129638671875 time spend to save output : 0.015079975128173828 total time spend for step 7 : 1.4280438423156738 step8:final Wed Mar 5 07:39:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1340952880: ('0.02336980028005479',), 1340952860: ('0.02336980028005479',), 1340952781: ('0.02336980028005479',), 1340952777: ('0.02336980028005479',), 1340952734: ('0.02336980028005479',), 1340952715: ('0.02336980028005479',), 1340952584: ('0.02336980028005479',), 1340952423: ('0.02336980028005479',), 1340952292: ('0.02336980028005479',), 1340952290: ('0.02336980028005479',)} new output for save of step final : {1340952880: ('0.02336980028005479',), 1340952860: ('0.02336980028005479',), 1340952781: ('0.02336980028005479',), 1340952777: ('0.02336980028005479',), 1340952734: ('0.02336980028005479',), 1340952715: ('0.02336980028005479',), 1340952584: ('0.02336980028005479',), 1340952423: ('0.02336980028005479',), 1340952292: ('0.02336980028005479',), 1340952290: ('0.02336980028005479',)} [1340952880, 1340952860, 1340952781, 1340952777, 1340952734, 1340952715, 1340952584, 1340952423, 1340952292, 1340952290] Looping around the photos to save general results len do output : 10 /1340952880.Didn't retrieve data . /1340952860.Didn't retrieve data . /1340952781.Didn't retrieve data . /1340952777.Didn't retrieve data . /1340952734.Didn't retrieve data . /1340952715.Didn't retrieve data . /1340952584.Didn't retrieve data . /1340952423.Didn't retrieve data . /1340952292.Didn't retrieve data . /1340952290.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952880', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952860', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952781', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952777', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952734', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952715', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952584', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952423', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952292', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952290', None, None, None, None, None, '2636165') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.012457132339477539 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.1101081371307373 time spend to save output : 0.012994050979614258 total time spend for step 8 : 0.12310218811035156 step9:velours_tree Wed Mar 5 07:39:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.02946162223815918 time spend to save output : 4.220008850097656e-05 total time spend for step 9 : 0.029503822326660156 step10:send_mail_cod Wed Mar 5 07:39:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21050245_05-03-2025_07_39_59.pdf 21050320 imagette210503201741156799 21050322 imagette210503221741156799 21050324 imagette210503241741156799 21050326 imagette210503261741156799 21050327 imagette210503271741156799 21050328 imagette210503281741156799 21050330 change filename to text .change filename to text .imagette210503301741156799 21050321 imagette210503211741156799 21050323 change filename to text .change filename to text .change filename to text .imagette210503231741156799 21050331 imagette210503311741156800 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21050245 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21050320,21050321,21050322,21050323,21050324,21050325,21050326,21050327,21050328,21050329,21050330,21050331?tags=metal,Carton_gris,kraft,Carton_brun,autre_refus,papier,plastique,mal_croppe,flou,environnement,cartonnette,Teint_Dans_La_Masse your option no_mail is active, we will not send the real mail to your client args[1340952880] : ('0.02336980028005479',) no score found for photo 1340952880 We are sending mail with results at report@fotonower.com args[1340952860] : ('0.02336980028005479',) no score found for photo 1340952860 We are sending mail with results at report@fotonower.com args[1340952781] : ('0.02336980028005479',) no score found for photo 1340952781 We are sending mail with results at report@fotonower.com args[1340952777] : ('0.02336980028005479',) no score found for photo 1340952777 We are sending mail with results at report@fotonower.com args[1340952734] : ('0.02336980028005479',) no score found for photo 1340952734 We are sending mail with results at report@fotonower.com args[1340952715] : ('0.02336980028005479',) no score found for photo 1340952715 We are sending mail with results at report@fotonower.com args[1340952584] : ('0.02336980028005479',) no score found for photo 1340952584 We are sending mail with results at report@fotonower.com args[1340952423] : ('0.02336980028005479',) no score found for photo 1340952423 We are sending mail with results at report@fotonower.com args[1340952292] : ('0.02336980028005479',) no score found for photo 1340952292 We are sending mail with results at report@fotonower.com args[1340952290] : ('0.02336980028005479',) no score found for photo 1340952290 We are sending mail with results at report@fotonower.com refus_total : 0.02336980028005479 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=21050245 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1340952166,1340952189) Found this number of photos: 2 begin to download photo : 1340952166 begin to download photo : 1340952189 download finish for photo 1340952189 download finish for photo 1340952166 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21050245_05-03-2025_07_39_59.pdf results_Auto_P21050245_05-03-2025_07_39_59.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21050245_05-03-2025_07_39_59.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','21050245','results_Auto_P21050245_05-03-2025_07_39_59.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21050245_05-03-2025_07_39_59.pdf','pdf','','0.07','0.02336980028005479') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1340952880, 1340952860, 1340952781, 1340952777, 1340952734, 1340952715, 1340952584, 1340952423, 1340952292, 1340952290] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952880', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952860', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952781', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952777', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952734', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952715', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952584', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952423', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952292', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952290', None, None, None, None, None, '2636165') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.013244390487670898 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.691554069519043 time spend to save output : 0.01349186897277832 total time spend for step 10 : 1.7050459384918213 step11:split_time_score Wed Mar 5 07:40:01 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('07', 22),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 05032025 21050245 Nombre de photos uploadées : 22 / 23040 (0%) 05032025 21050245 Nombre de photos taguées (types de déchets): 0 / 22 (0%) 05032025 21050245 Nombre de photos taguées (volume) : 0 / 22 (0%) elapsed_time : load_data_split_time_score 4.76837158203125e-06 elapsed_time : order_list_meta_photo_and_scores 1.049041748046875e-05 ?????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.001138448715209961 elapsed_time : insert_dashboard_record_day_entry 0.02364206314086914 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.013347235839879117 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21050245_05-03-2025_07_39_59.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21050245 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21050245 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'05032025': {'nb_upload': 22, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1340952880, 1340952860, 1340952781, 1340952777, 1340952734, 1340952715, 1340952584, 1340952423, 1340952292, 1340952290] Looping around the photos to save general results len do output : 1 /21050245Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952880', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952860', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952781', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952777', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952734', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952715', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952584', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952423', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952292', None, None, None, None, None, '2636165') ('3459', None, None, None, None, None, None, None, '2636165') ('3459', None, '1340952290', None, None, None, None, None, '2636165') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.015013694763183594 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.25162529945373535 time spend to save output : 0.01534128189086914 total time spend for step 11 : 0.2669665813446045 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 33.84user 21.80system 1:37.56elapsed 57%CPU (0avgtext+0avgdata 2706112maxresident)k 122568inputs+14056outputs (209major+2004097minor)pagefaults 0swaps