python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3775819 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 28>10 please execute split_portfolio.py -i 20915780 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2622651'] with mtr_portfolio_ids : ['20915780'] and first list_photo_ids : [] new path : /proc/3775819/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.1608977317810059 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 28 08:08:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-28 08:08:32.577254: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-28 08:08:32.611171: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-28 08:08:32.613630: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f6e98000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-28 08:08:32.613675: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-28 08:08:32.619855: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-28 08:08:32.756021: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x342a8560 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-28 08:08:32.756068: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-28 08:08:32.757403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 08:08:32.757769: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 08:08:32.760111: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 08:08:32.777025: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 08:08:32.777466: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 08:08:32.804724: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 08:08:32.810656: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 08:08:32.871531: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 08:08:32.873291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 08:08:32.873375: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 08:08:32.874317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-28 08:08:32.874342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-28 08:08:32.874357: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-28 08:08:32.876045: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4631 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-28 08:08:33.198425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 08:08:33.198548: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 08:08:33.198573: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 08:08:33.198594: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 08:08:33.198614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 08:08:33.198634: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 08:08:33.198654: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 08:08:33.198675: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 08:08:33.199887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 08:08:33.201203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 08:08:33.201275: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 08:08:33.201298: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 08:08:33.201319: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 08:08:33.201339: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 08:08:33.201358: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 08:08:33.201378: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 08:08:33.201398: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 08:08:33.202588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 08:08:33.202630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-28 08:08:33.202641: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-28 08:08:33.202651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-28 08:08:33.203935: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4631 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-28 08:08:42.998170: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 08:08:43.269909: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 08:08:45.107381: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 Detection mask done ! Trying to reset tf kernel 3776285 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 202 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5083 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.000164031982421875 nb_pixel_total : 3462 time to create 1 rle with old method : 0.004401683807373047 length of segment : 95 time for calcul the mask position with numpy : 0.0006053447723388672 nb_pixel_total : 37904 time to create 1 rle with old method : 0.04633021354675293 length of segment : 210 time for calcul the mask position with numpy : 0.0013496875762939453 nb_pixel_total : 57219 time to create 1 rle with old method : 0.0696709156036377 length of segment : 444 time for calcul the mask position with numpy : 0.008254528045654297 nb_pixel_total : 231973 time to create 1 rle with new method : 0.06324315071105957 length of segment : 1263 time for calcul the mask position with numpy : 0.00012373924255371094 nb_pixel_total : 2771 time to create 1 rle with old method : 0.003370523452758789 length of segment : 76 time for calcul the mask position with numpy : 0.004761934280395508 nb_pixel_total : 333828 time to create 1 rle with new method : 0.013117313385009766 length of segment : 1090 time for calcul the mask position with numpy : 0.000125885009765625 nb_pixel_total : 1897 time to create 1 rle with old method : 0.0025489330291748047 length of segment : 64 time for calcul the mask position with numpy : 0.001535177230834961 nb_pixel_total : 76239 time to create 1 rle with old method : 0.09143924713134766 length of segment : 728 time for calcul the mask position with numpy : 0.006465435028076172 nb_pixel_total : 418711 time to create 1 rle with new method : 0.012892961502075195 length of segment : 1315 time for calcul the mask position with numpy : 0.011710643768310547 nb_pixel_total : 834809 time to create 1 rle with new method : 0.03922224044799805 length of segment : 1525 time for calcul the mask position with numpy : 0.00013327598571777344 nb_pixel_total : 3987 time to create 1 rle with old method : 0.004931926727294922 length of segment : 94 time for calcul the mask position with numpy : 0.00021409988403320312 nb_pixel_total : 8606 time to create 1 rle with old method : 0.010235309600830078 length of segment : 186 time for calcul the mask position with numpy : 8.177757263183594e-05 nb_pixel_total : 1795 time to create 1 rle with old method : 0.0022597312927246094 length of segment : 64 time for calcul the mask position with numpy : 5.435943603515625e-05 nb_pixel_total : 1009 time to create 1 rle with old method : 0.001379251480102539 length of segment : 30 time for calcul the mask position with numpy : 0.004214763641357422 nb_pixel_total : 317691 time to create 1 rle with new method : 0.012247085571289062 length of segment : 934 time for calcul the mask position with numpy : 0.0010154247283935547 nb_pixel_total : 93323 time to create 1 rle with old method : 0.1311328411102295 length of segment : 206 time for calcul the mask position with numpy : 0.0005662441253662109 nb_pixel_total : 31349 time to create 1 rle with old method : 0.04112577438354492 length of segment : 137 time for calcul the mask position with numpy : 0.0002963542938232422 nb_pixel_total : 6662 time to create 1 rle with old method : 0.008086204528808594 length of segment : 66 time for calcul the mask position with numpy : 0.003702402114868164 nb_pixel_total : 148767 time to create 1 rle with old method : 0.16449570655822754 length of segment : 718 time for calcul the mask position with numpy : 0.000171661376953125 nb_pixel_total : 3197 time to create 1 rle with old method : 0.003919363021850586 length of segment : 50 time for calcul the mask position with numpy : 0.010291099548339844 nb_pixel_total : 429384 time to create 1 rle with new method : 0.048874855041503906 length of segment : 1772 time for calcul the mask position with numpy : 0.00032520294189453125 nb_pixel_total : 10387 time to create 1 rle with old method : 0.011623382568359375 length of segment : 102 time for calcul the mask position with numpy : 0.00027561187744140625 nb_pixel_total : 12545 time to create 1 rle with old method : 0.01405787467956543 length of segment : 109 time for calcul the mask position with numpy : 0.0015876293182373047 nb_pixel_total : 80500 time to create 1 rle with old method : 0.08996868133544922 length of segment : 532 time for calcul the mask position with numpy : 0.016238927841186523 nb_pixel_total : 854527 time to create 1 rle with new method : 0.165910005569458 length of segment : 1113 time for calcul the mask position with numpy : 0.001890420913696289 nb_pixel_total : 114927 time to create 1 rle with old method : 0.12981653213500977 length of segment : 983 time for calcul the mask position with numpy : 0.003742218017578125 nb_pixel_total : 315777 time to create 1 rle with new method : 0.009600162506103516 length of segment : 939 time for calcul the mask position with numpy : 0.000278472900390625 nb_pixel_total : 14143 time to create 1 rle with old method : 0.016283750534057617 length of segment : 204 time for calcul the mask position with numpy : 0.0015728473663330078 nb_pixel_total : 55178 time to create 1 rle with old method : 0.06650400161743164 length of segment : 519 time for calcul the mask position with numpy : 0.0007879734039306641 nb_pixel_total : 29900 time to create 1 rle with old method : 0.036164283752441406 length of segment : 268 time for calcul the mask position with numpy : 0.017662763595581055 nb_pixel_total : 853945 time to create 1 rle with new method : 0.09850645065307617 length of segment : 1364 time for calcul the mask position with numpy : 0.001371622085571289 nb_pixel_total : 88355 time to create 1 rle with old method : 0.10139036178588867 length of segment : 412 time for calcul the mask position with numpy : 0.0011858940124511719 nb_pixel_total : 103818 time to create 1 rle with old method : 0.11604118347167969 length of segment : 329 time for calcul the mask position with numpy : 0.0006008148193359375 nb_pixel_total : 21650 time to create 1 rle with old method : 0.02570033073425293 length of segment : 244 time for calcul the mask position with numpy : 0.008407354354858398 nb_pixel_total : 421800 time to create 1 rle with new method : 0.06659984588623047 length of segment : 1500 time for calcul the mask position with numpy : 0.0045032501220703125 nb_pixel_total : 298433 time to create 1 rle with new method : 0.010102033615112305 length of segment : 1117 time for calcul the mask position with numpy : 0.0003113746643066406 nb_pixel_total : 5522 time to create 1 rle with old method : 0.0072405338287353516 length of segment : 108 time for calcul the mask position with numpy : 0.0006213188171386719 nb_pixel_total : 18536 time to create 1 rle with old method : 0.03141069412231445 length of segment : 134 time for calcul the mask position with numpy : 0.013051509857177734 nb_pixel_total : 714552 time to create 1 rle with new method : 0.17459893226623535 length of segment : 1195 time for calcul the mask position with numpy : 0.0018062591552734375 nb_pixel_total : 126750 time to create 1 rle with old method : 0.14702534675598145 length of segment : 675 time for calcul the mask position with numpy : 0.0013988018035888672 nb_pixel_total : 80155 time to create 1 rle with old method : 0.09569048881530762 length of segment : 436 time for calcul the mask position with numpy : 0.00042819976806640625 nb_pixel_total : 22616 time to create 1 rle with old method : 0.026337862014770508 length of segment : 133 time for calcul the mask position with numpy : 0.0002644062042236328 nb_pixel_total : 13861 time to create 1 rle with old method : 0.01636958122253418 length of segment : 160 time for calcul the mask position with numpy : 0.001001119613647461 nb_pixel_total : 73245 time to create 1 rle with old method : 0.09088873863220215 length of segment : 372 time for calcul the mask position with numpy : 8.344650268554688e-05 nb_pixel_total : 2879 time to create 1 rle with old method : 0.0037238597869873047 length of segment : 43 time for calcul the mask position with numpy : 0.0005245208740234375 nb_pixel_total : 22868 time to create 1 rle with old method : 0.02694535255432129 length of segment : 136 time for calcul the mask position with numpy : 0.0003542900085449219 nb_pixel_total : 12047 time to create 1 rle with old method : 0.014519214630126953 length of segment : 136 time for calcul the mask position with numpy : 0.0011363029479980469 nb_pixel_total : 52422 time to create 1 rle with old method : 0.06017279624938965 length of segment : 282 time for calcul the mask position with numpy : 0.0014846324920654297 nb_pixel_total : 47401 time to create 1 rle with old method : 0.05391812324523926 length of segment : 485 time for calcul the mask position with numpy : 0.0003688335418701172 nb_pixel_total : 4930 time to create 1 rle with old method : 0.00606536865234375 length of segment : 184 time for calcul the mask position with numpy : 0.01013803482055664 nb_pixel_total : 368741 time to create 1 rle with new method : 0.03936600685119629 length of segment : 1059 time for calcul the mask position with numpy : 0.00022482872009277344 nb_pixel_total : 4192 time to create 1 rle with old method : 0.0051538944244384766 length of segment : 85 time for calcul the mask position with numpy : 0.0007996559143066406 nb_pixel_total : 29859 time to create 1 rle with old method : 0.03795003890991211 length of segment : 242 time for calcul the mask position with numpy : 0.011296987533569336 nb_pixel_total : 518532 time to create 1 rle with new method : 0.08806562423706055 length of segment : 1379 time for calcul the mask position with numpy : 0.00023174285888671875 nb_pixel_total : 11688 time to create 1 rle with old method : 0.01367497444152832 length of segment : 135 time spent for convertir_results : 9.118290424346924 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 55 chid ids of type : 3663 Number RLEs to save : 28181 save missing photos in datou_result : time spend for datou_step_exec : 35.88449430465698 time spend to save output : 1.6684377193450928 total time spend for step 1 : 37.552932024002075 step2:crop_condition Fri Feb 28 08:09:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 55 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340088288, 1340088211, 1340088204, 1340088157, 1340088156, 1340088113, 1340088106, 1340088043, 1340088038, 1340087983] Looping around the photos to save general results len do output : 16 /-3689713908Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713907Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713920Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713936Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713942Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713939Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713949Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713934Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713904Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713925Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713935Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713944Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713945Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713943Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713932Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689713919Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088288', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088211', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088204', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088157', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088156', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088113', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088106', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088043', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088038', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340087983', None, None, None, None, None, '2622651') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 58 time used for this insertion : 0.019016742706298828 save_final save missing photos in datou_result : time spend for datou_step_exec : 5.30843448638916 time spend to save output : 0.020158052444458008 total time spend for step 2 : 5.328592538833618 step3:thcl Fri Feb 28 08:09:11 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.008170604705810547 time to convert the images to numpy array : 1.9073486328125e-06 time to import caffe and check if the image exist : 0.008324623107910156 time to convert the images to numpy array : 2.6226043701171875e-06 time to import caffe and check if the image exist : 0.00684666633605957 time to convert the images to numpy array : 0.010085582733154297 time to import caffe and check if the image exist : 0.008122682571411133 time to convert the images to numpy array : 0.010162115097045898 time to import caffe and check if the image exist : 0.009053230285644531 time to convert the images to numpy array : 0.011089563369750977 time to import caffe and check if the image exist : 0.00957179069519043 time to convert the images to numpy array : 0.015302896499633789 time to import caffe and check if the image exist : 0.010014533996582031 time to convert the images to numpy array : 0.014879703521728516 time to import caffe and check if the image exist : 0.010307073593139648 time to convert the images to numpy array : 0.019453048706054688 time to import caffe and check if the image exist : 0.011315107345581055 time to convert the images to numpy array : 0.018806934356689453 time to import caffe and check if the image exist : 0.0074956417083740234 time to convert the images to numpy array : 0.0589299201965332 total time to convert the images to numpy array : 0.2576906681060791 list photo_ids error: [] list photo_ids correct : [-3689713920, -3689713936, -3689713904, -3689713925, -3689713932, -3689713919, -3689713935, -3689713944, -3689713945, -3689713943, -3689713942, -3689713939, -3689713949, -3689713934, -3689713908, -3689713907] number of photos to traite : 16 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4864 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.188232421875 time used to do the prediction : 0.08276152610778809 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4862 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4862 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.20679068565368652 time used to do the prediction : 0.05884742736816406 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.0967254638671875e-05 save missing photos in datou_result : time spend for datou_step_exec : 7.707694053649902 time spend to save output : 0.00021886825561523438 total time spend for step 3 : 7.707912921905518 step4:merge_mask_thcl_custom Fri Feb 28 08:09:18 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 55 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1340088288, 1340088211, 1340088204, 1340088157, 1340088156, 1340088113, 1340088106, 1340088043, 1340088038, 1340087983] Looping around the photos to save general results len do output : 10 /1340088288Didn't retrieve data .Didn't retrieve data . /1340088211Didn't retrieve data .Didn't retrieve data . /1340088204Didn't retrieve data .Didn't retrieve data . /1340088157Didn't retrieve data .Didn't retrieve data . /1340088156Didn't retrieve data .Didn't retrieve data . /1340088113Didn't retrieve data .Didn't retrieve data . /1340088106Didn't retrieve data .Didn't retrieve data . /1340088043Didn't retrieve data .Didn't retrieve data . /1340088038Didn't retrieve data .Didn't retrieve data . /1340087983Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088288', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088211', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088204', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088157', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088156', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088113', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088106', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088043', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088038', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340087983', None, None, None, None, None, '2622651') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.0131683349609375 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.04738450050354004 time spend to save output : 0.013700485229492188 total time spend for step 4 : 0.06108498573303223 step5:rle_unique_nms_with_priority Fri Feb 28 08:09:18 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 1.4803440570831299 time for calcul the mask position with numpy : 0.29446840286254883 nb_pixel_total : 1605056 time to create 1 rle with new method : 0.2663702964782715 time for calcul the mask position with numpy : 0.008028268814086914 nb_pixel_total : 132834 time to create 1 rle with old method : 0.14674711227416992 time for calcul the mask position with numpy : 0.006625175476074219 nb_pixel_total : 1862 time to create 1 rle with old method : 0.0022928714752197266 time for calcul the mask position with numpy : 0.00632023811340332 nb_pixel_total : 2759 time to create 1 rle with old method : 0.003159761428833008 time for calcul the mask position with numpy : 0.007709503173828125 nb_pixel_total : 232753 time to create 1 rle with new method : 0.10365867614746094 time for calcul the mask position with numpy : 0.007022857666015625 nb_pixel_total : 57031 time to create 1 rle with old method : 0.06480979919433594 time for calcul the mask position with numpy : 0.006624698638916016 nb_pixel_total : 3448 time to create 1 rle with old method : 0.004088878631591797 time for calcul the mask position with numpy : 0.006871223449707031 nb_pixel_total : 37857 time to create 1 rle with old method : 0.04628419876098633 create new chi : 0.9994485378265381 time to delete rle : 0.022478818893432617 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6729 TO DO : save crop sub photo not yet done ! save time : 0.4167180061340332 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 0.2982912063598633 time for calcul the mask position with numpy : 0.01697087287902832 nb_pixel_total : 1635061 time to create 1 rle with new method : 0.3405494689941406 time for calcul the mask position with numpy : 0.009897708892822266 nb_pixel_total : 362472 time to create 1 rle with new method : 0.0364072322845459 time for calcul the mask position with numpy : 0.0071294307708740234 nb_pixel_total : 76067 time to create 1 rle with old method : 0.08624553680419922 create new chi : 0.49796128273010254 time to delete rle : 0.00043463706970214844 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 4051 TO DO : save crop sub photo not yet done ! save time : 0.2573678493499756 nb_obj : 7 nb_hashtags : 2 time to prepare the origin masks : 0.7109375 time for calcul the mask position with numpy : 0.023302078247070312 nb_pixel_total : 1131003 time to create 1 rle with new method : 0.1558513641357422 time for calcul the mask position with numpy : 0.008397102355957031 nb_pixel_total : 31287 time to create 1 rle with old method : 0.04147696495056152 time for calcul the mask position with numpy : 0.008249521255493164 nb_pixel_total : 52315 time to create 1 rle with old method : 0.06304383277893066 time for calcul the mask position with numpy : 0.0071468353271484375 nb_pixel_total : 1005 time to create 1 rle with old method : 0.0011777877807617188 time for calcul the mask position with numpy : 0.006886005401611328 nb_pixel_total : 1787 time to create 1 rle with old method : 0.0020301342010498047 time for calcul the mask position with numpy : 0.006879568099975586 nb_pixel_total : 8550 time to create 1 rle with old method : 0.010207653045654297 time for calcul the mask position with numpy : 0.00704193115234375 nb_pixel_total : 3969 time to create 1 rle with old method : 0.0045816898345947266 time for calcul the mask position with numpy : 0.014953374862670898 nb_pixel_total : 843684 time to create 1 rle with new method : 0.08506965637207031 create new chi : 0.4645869731903076 time to delete rle : 0.000568389892578125 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 5435 TO DO : save crop sub photo not yet done ! save time : 0.3231234550476074 nb_obj : 6 nb_hashtags : 3 time to prepare the origin masks : 0.550412654876709 time for calcul the mask position with numpy : 0.25536561012268066 nb_pixel_total : 1438143 time to create 1 rle with new method : 0.12676143646240234 time for calcul the mask position with numpy : 0.007552623748779297 nb_pixel_total : 12505 time to create 1 rle with old method : 0.014414072036743164 time for calcul the mask position with numpy : 0.007269144058227539 nb_pixel_total : 10381 time to create 1 rle with old method : 0.011756181716918945 time for calcul the mask position with numpy : 0.012459039688110352 nb_pixel_total : 454200 time to create 1 rle with new method : 0.3938257694244385 time for calcul the mask position with numpy : 0.006582736968994141 nb_pixel_total : 6571 time to create 1 rle with old method : 0.007788419723510742 time for calcul the mask position with numpy : 0.006455659866333008 nb_pixel_total : 3166 time to create 1 rle with old method : 0.003575563430786133 time for calcul the mask position with numpy : 0.007001399993896484 nb_pixel_total : 148634 time to create 1 rle with old method : 0.21314311027526855 create new chi : 1.0956199169158936 time to delete rle : 0.0007617473602294922 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 4622 TO DO : save crop sub photo not yet done ! save time : 0.3043074607849121 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.573272705078125 time for calcul the mask position with numpy : 0.015427350997924805 nb_pixel_total : 1064583 time to create 1 rle with new method : 0.33400797843933105 time for calcul the mask position with numpy : 0.007219076156616211 nb_pixel_total : 8386 time to create 1 rle with old method : 0.01361083984375 time for calcul the mask position with numpy : 0.006926774978637695 nb_pixel_total : 31178 time to create 1 rle with old method : 0.040006399154663086 time for calcul the mask position with numpy : 0.0070438385009765625 nb_pixel_total : 114743 time to create 1 rle with old method : 0.1295180320739746 time for calcul the mask position with numpy : 0.012479543685913086 nb_pixel_total : 854710 time to create 1 rle with new method : 0.031042814254760742 create new chi : 0.5985138416290283 time to delete rle : 0.0006220340728759766 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 6395 TO DO : save crop sub photo not yet done ! save time : 0.4015998840332031 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 1.0121173858642578 time for calcul the mask position with numpy : 0.045018911361694336 nb_pixel_total : 975074 time to create 1 rle with new method : 0.11700606346130371 time for calcul the mask position with numpy : 0.007625579833984375 nb_pixel_total : 50903 time to create 1 rle with old method : 0.058470726013183594 time for calcul the mask position with numpy : 0.008594036102294922 nb_pixel_total : 103603 time to create 1 rle with old method : 0.1112060546875 time for calcul the mask position with numpy : 0.00789022445678711 nb_pixel_total : 28868 time to create 1 rle with old method : 0.031470298767089844 time for calcul the mask position with numpy : 0.017389297485351562 nb_pixel_total : 860146 time to create 1 rle with new method : 0.33165860176086426 time for calcul the mask position with numpy : 0.007289886474609375 nb_pixel_total : 55006 time to create 1 rle with old method : 0.06256556510925293 create new chi : 0.8270344734191895 time to delete rle : 0.0006716251373291016 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6195 TO DO : save crop sub photo not yet done ! save time : 0.3711273670196533 nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 0.43262195587158203 time for calcul the mask position with numpy : 0.022760629653930664 nb_pixel_total : 1532088 time to create 1 rle with new method : 0.14664840698242188 time for calcul the mask position with numpy : 0.006971597671508789 nb_pixel_total : 21577 time to create 1 rle with old method : 0.022878646850585938 time for calcul the mask position with numpy : 0.007412433624267578 nb_pixel_total : 98587 time to create 1 rle with old method : 0.10683155059814453 time for calcul the mask position with numpy : 0.01069951057434082 nb_pixel_total : 421348 time to create 1 rle with new method : 0.2618846893310547 create new chi : 0.5956301689147949 time to delete rle : 0.0005934238433837891 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 5647 TO DO : save crop sub photo not yet done ! save time : 0.33792924880981445 nb_obj : 9 nb_hashtags : 3 time to prepare the origin masks : 1.5160765647888184 time for calcul the mask position with numpy : 0.03912782669067383 nb_pixel_total : 1087455 time to create 1 rle with new method : 0.3231644630432129 time for calcul the mask position with numpy : 0.006329059600830078 nb_pixel_total : 13799 time to create 1 rle with old method : 0.015161752700805664 time for calcul the mask position with numpy : 0.01025533676147461 nb_pixel_total : 2833 time to create 1 rle with old method : 0.0032949447631835938 time for calcul the mask position with numpy : 0.008906126022338867 nb_pixel_total : 2997 time to create 1 rle with old method : 0.0038461685180664062 time for calcul the mask position with numpy : 0.009774208068847656 nb_pixel_total : 22507 time to create 1 rle with old method : 0.026079893112182617 time for calcul the mask position with numpy : 0.0070497989654541016 nb_pixel_total : 76467 time to create 1 rle with old method : 0.08739089965820312 time for calcul the mask position with numpy : 0.016848087310791016 nb_pixel_total : 717881 time to create 1 rle with new method : 0.11293244361877441 time for calcul the mask position with numpy : 0.006572246551513672 nb_pixel_total : 5501 time to create 1 rle with old method : 0.006136178970336914 time for calcul the mask position with numpy : 0.007071733474731445 nb_pixel_total : 18450 time to create 1 rle with old method : 0.02317070960998535 time for calcul the mask position with numpy : 0.008905887603759766 nb_pixel_total : 125710 time to create 1 rle with old method : 0.1659984588623047 create new chi : 0.8977177143096924 time to delete rle : 0.0009748935699462891 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 6648 TO DO : save crop sub photo not yet done ! save time : 0.4246397018432617 nb_obj : 6 nb_hashtags : 2 time to prepare the origin masks : 0.641756534576416 time for calcul the mask position with numpy : 0.022932767868041992 nb_pixel_total : 1585022 time to create 1 rle with new method : 0.037835121154785156 time for calcul the mask position with numpy : 0.008330583572387695 nb_pixel_total : 349321 time to create 1 rle with new method : 0.02974677085876465 time for calcul the mask position with numpy : 0.006597995758056641 nb_pixel_total : 4907 time to create 1 rle with old method : 0.005657196044921875 time for calcul the mask position with numpy : 0.006305694580078125 nb_pixel_total : 12011 time to create 1 rle with old method : 0.013196945190429688 time for calcul the mask position with numpy : 0.006289005279541016 nb_pixel_total : 47301 time to create 1 rle with old method : 0.05237245559692383 time for calcul the mask position with numpy : 0.006722211837768555 nb_pixel_total : 52258 time to create 1 rle with old method : 0.05928850173950195 time for calcul the mask position with numpy : 0.006715536117553711 nb_pixel_total : 22780 time to create 1 rle with old method : 0.02813720703125 create new chi : 0.29764604568481445 time to delete rle : 0.0008797645568847656 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 5467 TO DO : save crop sub photo not yet done ! save time : 0.3334634304046631 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.3968038558959961 time for calcul the mask position with numpy : 0.10948300361633301 nb_pixel_total : 1475335 time to create 1 rle with new method : 0.10482573509216309 time for calcul the mask position with numpy : 0.006685733795166016 nb_pixel_total : 11648 time to create 1 rle with old method : 0.013579130172729492 time for calcul the mask position with numpy : 0.010904312133789062 nb_pixel_total : 552668 time to create 1 rle with new method : 0.285205602645874 time for calcul the mask position with numpy : 0.0075457096099853516 nb_pixel_total : 29773 time to create 1 rle with old method : 0.033768653869628906 time for calcul the mask position with numpy : 0.0063018798828125 nb_pixel_total : 4176 time to create 1 rle with old method : 0.004496097564697266 create new chi : 0.5919349193572998 time to delete rle : 0.0004100799560546875 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 4184 TO DO : save crop sub photo not yet done ! save time : 0.2692286968231201 map_output_result : {1340088288: (0.0, 'Should be the crop_list due to order', 0), 1340088211: (0.0, 'Should be the crop_list due to order', 0), 1340088204: (0.0, 'Should be the crop_list due to order', 0), 1340088157: (0.0, 'Should be the crop_list due to order', 0), 1340088156: (0.0, 'Should be the crop_list due to order', 0), 1340088113: (0.0, 'Should be the crop_list due to order', 0), 1340088106: (0.0, 'Should be the crop_list due to order', 0), 1340088043: (0.0, 'Should be the crop_list due to order', 0), 1340088038: (0.0, 'Should be the crop_list due to order', 0), 1340087983: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1340088288, 1340088211, 1340088204, 1340088157, 1340088156, 1340088113, 1340088106, 1340088043, 1340088038, 1340087983] Looping around the photos to save general results len do output : 10 /1340088288.Didn't retrieve data . /1340088211.Didn't retrieve data . /1340088204.Didn't retrieve data . /1340088157.Didn't retrieve data . /1340088156.Didn't retrieve data . /1340088113.Didn't retrieve data . /1340088106.Didn't retrieve data . /1340088043.Didn't retrieve data . /1340088038.Didn't retrieve data . /1340087983.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088288', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088211', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088204', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088157', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088156', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088113', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088106', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088043', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088038', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340087983', None, None, None, None, None, '2622651') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01294851303100586 save_final save missing photos in datou_result : time spend for datou_step_exec : 18.295861959457397 time spend to save output : 0.013447761535644531 total time spend for step 5 : 18.309309720993042 step6:crop_condition Fri Feb 28 08:09:37 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 63 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740726578_3775819 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.762880802154541 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740726579_3775819 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6577630043029785 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740726580_3775819 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.0977833271026611 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740726582_3775819 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6315481662750244 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740726583_3775819 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8529572486877441 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340088288, 1340088211, 1340088204, 1340088157, 1340088156, 1340088113, 1340088106, 1340088043, 1340088038, 1340087983] Looping around the photos to save general results len do output : 7 /1340091794Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340091795Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340091796Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340091798Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340091799Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340091801Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340091804Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088288', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088211', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088204', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088157', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088156', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088113', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088106', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088043', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088038', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340087983', None, None, None, None, None, '2622651') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 31 time used for this insertion : 0.02517390251159668 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.026549816131592 time spend to save output : 0.025649070739746094 total time spend for step 6 : 7.052198886871338 step7:ventilate_hashtags_in_portfolio Fri Feb 28 08:09:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20915780 get user id for portfolio 20915780 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20915780 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('autre_refus','Teint_Dans_La_Masse','kraft','environnement','Carton_brun','mal_croppe','flou','papier','Carton_gris','plastique','metal','cartonnette')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20915780 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('autre_refus','Teint_Dans_La_Masse','kraft','environnement','Carton_brun','mal_croppe','flou','papier','Carton_gris','plastique','metal','cartonnette')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20915780 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('autre_refus','Teint_Dans_La_Masse','kraft','environnement','Carton_brun','mal_croppe','flou','papier','Carton_gris','plastique','metal','cartonnette')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20915933,20915934,20915935,20915936,20915937,20915938,20915939,20915940,20915941,20915942,20915943,20915944?tags=autre_refus,Teint_Dans_La_Masse,kraft,environnement,Carton_brun,mal_croppe,flou,papier,Carton_gris,plastique,metal,cartonnette Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1340088288, 1340088211, 1340088204, 1340088157, 1340088156, 1340088113, 1340088106, 1340088043, 1340088038, 1340087983] Looping around the photos to save general results len do output : 1 /20915780. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088288', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088211', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088204', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088157', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088156', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088113', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088106', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088043', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088038', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340087983', None, None, None, None, None, '2622651') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013747692108154297 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.6387431621551514 time spend to save output : 0.014045476913452148 total time spend for step 7 : 1.6527886390686035 step8:final Fri Feb 28 08:09:45 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1340088288: ('0.03410941429420713',), 1340088211: ('0.03410941429420713',), 1340088204: ('0.03410941429420713',), 1340088157: ('0.03410941429420713',), 1340088156: ('0.03410941429420713',), 1340088113: ('0.03410941429420713',), 1340088106: ('0.03410941429420713',), 1340088043: ('0.03410941429420713',), 1340088038: ('0.03410941429420713',), 1340087983: ('0.03410941429420713',)} new output for save of step final : {1340088288: ('0.03410941429420713',), 1340088211: ('0.03410941429420713',), 1340088204: ('0.03410941429420713',), 1340088157: ('0.03410941429420713',), 1340088156: ('0.03410941429420713',), 1340088113: ('0.03410941429420713',), 1340088106: ('0.03410941429420713',), 1340088043: ('0.03410941429420713',), 1340088038: ('0.03410941429420713',), 1340087983: ('0.03410941429420713',)} [1340088288, 1340088211, 1340088204, 1340088157, 1340088156, 1340088113, 1340088106, 1340088043, 1340088038, 1340087983] Looping around the photos to save general results len do output : 10 /1340088288.Didn't retrieve data . /1340088211.Didn't retrieve data . /1340088204.Didn't retrieve data . /1340088157.Didn't retrieve data . /1340088156.Didn't retrieve data . /1340088113.Didn't retrieve data . /1340088106.Didn't retrieve data . /1340088043.Didn't retrieve data . /1340088038.Didn't retrieve data . /1340087983.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088288', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088211', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088204', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088157', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088156', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088113', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088106', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088043', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088038', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340087983', None, None, None, None, None, '2622651') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01279306411743164 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.1115574836730957 time spend to save output : 0.013264179229736328 total time spend for step 8 : 0.12482166290283203 step9:velours_tree Fri Feb 28 08:09:45 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.03663825988769531 time spend to save output : 3.337860107421875e-05 total time spend for step 9 : 0.03667163848876953 step10:send_mail_cod Fri Feb 28 08:09:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20915780_28-02-2025_08_09_46.pdf 20915933 imagette209159331740726586 20915935 change filename to text .imagette209159351740726586 20915938 imagette209159381740726586 20915939 imagette209159391740726586 20915942 change filename to text .imagette209159421740726586 20915943 imagette209159431740726586 20915944 change filename to text .imagette209159441740726586 20915934 change filename to text .change filename to text .imagette209159341740726586 20915937 change filename to text .change filename to text .imagette209159371740726586 20915941 imagette209159411740726586 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20915780 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20915933,20915934,20915935,20915936,20915937,20915938,20915939,20915940,20915941,20915942,20915943,20915944?tags=autre_refus,Teint_Dans_La_Masse,kraft,environnement,Carton_brun,mal_croppe,flou,papier,Carton_gris,plastique,metal,cartonnette your option no_mail is active, we will not send the real mail to your client args[1340088288] : ('0.03410941429420713',) no score found for photo 1340088288 We are sending mail with results at report@fotonower.com args[1340088211] : ('0.03410941429420713',) no score found for photo 1340088211 We are sending mail with results at report@fotonower.com args[1340088204] : ('0.03410941429420713',) no score found for photo 1340088204 We are sending mail with results at report@fotonower.com args[1340088157] : ('0.03410941429420713',) no score found for photo 1340088157 We are sending mail with results at report@fotonower.com args[1340088156] : ('0.03410941429420713',) no score found for photo 1340088156 We are sending mail with results at report@fotonower.com args[1340088113] : ('0.03410941429420713',) no score found for photo 1340088113 We are sending mail with results at report@fotonower.com args[1340088106] : ('0.03410941429420713',) no score found for photo 1340088106 We are sending mail with results at report@fotonower.com args[1340088043] : ('0.03410941429420713',) no score found for photo 1340088043 We are sending mail with results at report@fotonower.com args[1340088038] : ('0.03410941429420713',) no score found for photo 1340088038 We are sending mail with results at report@fotonower.com args[1340087983] : ('0.03410941429420713',) no score found for photo 1340087983 We are sending mail with results at report@fotonower.com refus_total : 0.03410941429420713 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20915780 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1340086202,1340085974,1340086124,1340086051,1340086054,1340086120,1340086294,1340087166,1340087171) Found this number of photos: 9 begin to download photo : 1340086202 begin to download photo : 1340086124 begin to download photo : 1340086054 begin to download photo : 1340086294 begin to download photo : 1340087171 download finish for photo 1340087171 download finish for photo 1340086124 begin to download photo : 1340086051 download finish for photo 1340086202 begin to download photo : 1340085974 download finish for photo 1340086054 begin to download photo : 1340086120 download finish for photo 1340086294 begin to download photo : 1340087166 download finish for photo 1340086051 download finish for photo 1340085974 download finish for photo 1340087166 download finish for photo 1340086120 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20915780_28-02-2025_08_09_46.pdf results_Auto_P20915780_28-02-2025_08_09_46.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20915780_28-02-2025_08_09_46.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20915780','results_Auto_P20915780_28-02-2025_08_09_46.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20915780_28-02-2025_08_09_46.pdf','pdf','','0.13','0.03410941429420713') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1340088288, 1340088211, 1340088204, 1340088157, 1340088156, 1340088113, 1340088106, 1340088043, 1340088038, 1340087983] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088288', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088211', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088204', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088157', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088156', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088113', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088106', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088043', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088038', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340087983', None, None, None, None, None, '2622651') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.013208627700805664 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.8001558780670166 time spend to save output : 0.013414382934570312 total time spend for step 10 : 1.813570261001587 step11:split_time_score Fri Feb 28 08:09:47 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('07', 28),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 28022025 20915780 Nombre de photos uploadées : 28 / 23040 (0%) 28022025 20915780 Nombre de photos taguées (types de déchets): 0 / 28 (0%) 28022025 20915780 Nombre de photos taguées (volume) : 0 / 28 (0%) elapsed_time : load_data_split_time_score 3.814697265625e-06 elapsed_time : order_list_meta_photo_and_scores 8.344650268554688e-06 ???????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0011608600616455078 elapsed_time : insert_dashboard_record_day_entry 0.021596670150756836 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.05088624831321179 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20914260_28-02-2025_07_41_57.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20914260 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20914260 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20915778 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20915779 order by id desc limit 1 Qualite : 0.03410941429420713 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20915780_28-02-2025_08_09_46.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20915780 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20915780 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'28022025': {'nb_upload': 28, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1340088288, 1340088211, 1340088204, 1340088157, 1340088156, 1340088113, 1340088106, 1340088043, 1340088038, 1340087983] Looping around the photos to save general results len do output : 1 /20915780Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088288', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088211', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088204', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088157', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088156', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088113', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088106', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088043', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340088038', None, None, None, None, None, '2622651') ('3459', None, None, None, None, None, None, None, '2622651') ('3459', '20915780', '1340087983', None, None, None, None, None, '2622651') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013088226318359375 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.8376834392547607 time spend to save output : 0.013332843780517578 total time spend for step 11 : 1.8510162830352783 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 42.09user 31.99system 1:25.40elapsed 86%CPU (0avgtext+0avgdata 2762056maxresident)k 2171616inputs+21904outputs (7227major+2088929minor)pagefaults 0swaps