python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2865094 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 24>10 please execute split_portfolio.py -i 20503166 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2581919'] with mtr_portfolio_ids : ['20503166'] and first list_photo_ids : [] new path : /proc/2865094/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.133659839630127 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Thu Feb 13 08:26:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5043 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-13 08:26:33.831753: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-13 08:26:33.859128: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-13 08:26:33.861624: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fddb0000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-13 08:26:33.861724: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-13 08:26:33.866831: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-13 08:26:34.025744: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x124dee60 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-13 08:26:34.025825: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-13 08:26:34.027094: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-13 08:26:34.027765: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-13 08:26:34.032354: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-13 08:26:34.036181: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-13 08:26:34.037038: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-13 08:26:34.041494: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-13 08:26:34.043336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-13 08:26:34.051345: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-13 08:26:34.053196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-13 08:26:34.053434: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-13 08:26:34.054335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-13 08:26:34.054362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-13 08:26:34.054374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-13 08:26:34.055800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4591 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-13 08:26:34.403859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-13 08:26:34.404015: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-13 08:26:34.404055: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-13 08:26:34.404093: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-13 08:26:34.404121: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-13 08:26:34.404145: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-13 08:26:34.404171: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-13 08:26:34.404202: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-13 08:26:34.405422: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-13 08:26:34.406826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-13 08:26:34.406950: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-13 08:26:34.406998: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-13 08:26:34.407029: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-13 08:26:34.407057: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-13 08:26:34.407085: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-13 08:26:34.407113: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-13 08:26:34.407140: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-13 08:26:34.408335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-13 08:26:34.408395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-13 08:26:34.408410: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-13 08:26:34.408421: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-13 08:26:34.409817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4591 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-13 08:26:43.357958: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-13 08:26:43.557960: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-13 08:26:45.288682: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.98G (4279107584 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 Detection mask done ! Trying to reset tf kernel 2865366 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 397 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10553 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.002147197723388672 nb_pixel_total : 74288 time to create 1 rle with old method : 0.08017826080322266 length of segment : 305 time for calcul the mask position with numpy : 0.0004296302795410156 nb_pixel_total : 7257 time to create 1 rle with old method : 0.012372732162475586 length of segment : 85 time for calcul the mask position with numpy : 0.0003132820129394531 nb_pixel_total : 7403 time to create 1 rle with old method : 0.009024381637573242 length of segment : 108 time for calcul the mask position with numpy : 0.00043582916259765625 nb_pixel_total : 5564 time to create 1 rle with old method : 0.00663304328918457 length of segment : 237 time for calcul the mask position with numpy : 0.015440940856933594 nb_pixel_total : 522482 time to create 1 rle with new method : 0.03420257568359375 length of segment : 1511 time for calcul the mask position with numpy : 0.00030803680419921875 nb_pixel_total : 17562 time to create 1 rle with old method : 0.023355722427368164 length of segment : 227 time for calcul the mask position with numpy : 0.00020575523376464844 nb_pixel_total : 11303 time to create 1 rle with old method : 0.019311904907226562 length of segment : 60 time for calcul the mask position with numpy : 0.0017588138580322266 nb_pixel_total : 88862 time to create 1 rle with old method : 0.09806990623474121 length of segment : 547 time for calcul the mask position with numpy : 0.0005991458892822266 nb_pixel_total : 44436 time to create 1 rle with old method : 0.07544612884521484 length of segment : 287 time for calcul the mask position with numpy : 0.0011217594146728516 nb_pixel_total : 58907 time to create 1 rle with old method : 0.06395316123962402 length of segment : 291 time for calcul the mask position with numpy : 0.00025582313537597656 nb_pixel_total : 7120 time to create 1 rle with old method : 0.00865483283996582 length of segment : 87 time for calcul the mask position with numpy : 0.00037932395935058594 nb_pixel_total : 5451 time to create 1 rle with old method : 0.006258964538574219 length of segment : 221 time for calcul the mask position with numpy : 0.01107931137084961 nb_pixel_total : 449717 time to create 1 rle with new method : 0.018788814544677734 length of segment : 1497 time for calcul the mask position with numpy : 0.0001938343048095703 nb_pixel_total : 7792 time to create 1 rle with old method : 0.009032487869262695 length of segment : 120 time for calcul the mask position with numpy : 0.00024271011352539062 nb_pixel_total : 17095 time to create 1 rle with old method : 0.019724369049072266 length of segment : 218 time for calcul the mask position with numpy : 0.0006144046783447266 nb_pixel_total : 47799 time to create 1 rle with old method : 0.053865909576416016 length of segment : 295 time for calcul the mask position with numpy : 0.001188516616821289 nb_pixel_total : 85600 time to create 1 rle with old method : 0.09272098541259766 length of segment : 435 time for calcul the mask position with numpy : 0.0006511211395263672 nb_pixel_total : 26836 time to create 1 rle with old method : 0.03151679039001465 length of segment : 177 time for calcul the mask position with numpy : 0.0004761219024658203 nb_pixel_total : 11952 time to create 1 rle with old method : 0.01382589340209961 length of segment : 169 time for calcul the mask position with numpy : 0.016786575317382812 nb_pixel_total : 1084892 time to create 1 rle with new method : 0.03297615051269531 length of segment : 2000 time for calcul the mask position with numpy : 0.0028541088104248047 nb_pixel_total : 207376 time to create 1 rle with new method : 0.007332563400268555 length of segment : 666 time for calcul the mask position with numpy : 0.004297733306884766 nb_pixel_total : 374721 time to create 1 rle with new method : 0.011181831359863281 length of segment : 1728 time for calcul the mask position with numpy : 0.0015423297882080078 nb_pixel_total : 50400 time to create 1 rle with old method : 0.05568647384643555 length of segment : 622 time for calcul the mask position with numpy : 0.00374603271484375 nb_pixel_total : 147125 time to create 1 rle with old method : 0.17451739311218262 length of segment : 659 time for calcul the mask position with numpy : 0.017180204391479492 nb_pixel_total : 928562 time to create 1 rle with new method : 0.03250861167907715 length of segment : 1566 time for calcul the mask position with numpy : 0.0003857612609863281 nb_pixel_total : 18248 time to create 1 rle with old method : 0.019816875457763672 length of segment : 318 time for calcul the mask position with numpy : 0.001226186752319336 nb_pixel_total : 77158 time to create 1 rle with old method : 0.09076499938964844 length of segment : 563 time for calcul the mask position with numpy : 0.00014472007751464844 nb_pixel_total : 4498 time to create 1 rle with old method : 0.00521540641784668 length of segment : 84 time for calcul the mask position with numpy : 0.0012195110321044922 nb_pixel_total : 77186 time to create 1 rle with old method : 0.08563470840454102 length of segment : 265 time for calcul the mask position with numpy : 0.0004096031188964844 nb_pixel_total : 18350 time to create 1 rle with old method : 0.020671606063842773 length of segment : 208 time for calcul the mask position with numpy : 0.00011658668518066406 nb_pixel_total : 4871 time to create 1 rle with old method : 0.006136655807495117 length of segment : 42 time for calcul the mask position with numpy : 0.00033736228942871094 nb_pixel_total : 24759 time to create 1 rle with old method : 0.027407407760620117 length of segment : 189 time for calcul the mask position with numpy : 0.004041433334350586 nb_pixel_total : 311317 time to create 1 rle with new method : 0.013159036636352539 length of segment : 1025 time for calcul the mask position with numpy : 0.015506505966186523 nb_pixel_total : 967261 time to create 1 rle with new method : 0.034409523010253906 length of segment : 1771 time for calcul the mask position with numpy : 0.0006775856018066406 nb_pixel_total : 42886 time to create 1 rle with old method : 0.047788143157958984 length of segment : 232 time for calcul the mask position with numpy : 0.0027480125427246094 nb_pixel_total : 194010 time to create 1 rle with new method : 0.007750511169433594 length of segment : 992 time for calcul the mask position with numpy : 0.002961874008178711 nb_pixel_total : 238068 time to create 1 rle with new method : 0.007750034332275391 length of segment : 787 time for calcul the mask position with numpy : 0.0037488937377929688 nb_pixel_total : 308500 time to create 1 rle with new method : 0.008946657180786133 length of segment : 1031 time for calcul the mask position with numpy : 0.01344609260559082 nb_pixel_total : 684453 time to create 1 rle with new method : 0.03148460388183594 length of segment : 1311 time for calcul the mask position with numpy : 0.00015234947204589844 nb_pixel_total : 5389 time to create 1 rle with old method : 0.006432533264160156 length of segment : 125 time for calcul the mask position with numpy : 0.00026798248291015625 nb_pixel_total : 17507 time to create 1 rle with old method : 0.019468307495117188 length of segment : 179 time for calcul the mask position with numpy : 0.002132415771484375 nb_pixel_total : 143381 time to create 1 rle with old method : 0.16151833534240723 length of segment : 618 time for calcul the mask position with numpy : 0.0036499500274658203 nb_pixel_total : 338492 time to create 1 rle with new method : 0.008090019226074219 length of segment : 1071 time for calcul the mask position with numpy : 0.0014154911041259766 nb_pixel_total : 61542 time to create 1 rle with old method : 0.09492158889770508 length of segment : 357 time for calcul the mask position with numpy : 0.0001990795135498047 nb_pixel_total : 4623 time to create 1 rle with old method : 0.005484819412231445 length of segment : 70 time for calcul the mask position with numpy : 0.01215052604675293 nb_pixel_total : 612731 time to create 1 rle with new method : 0.02280449867248535 length of segment : 1293 time for calcul the mask position with numpy : 0.0009818077087402344 nb_pixel_total : 74294 time to create 1 rle with old method : 0.0828864574432373 length of segment : 328 time for calcul the mask position with numpy : 0.0019369125366210938 nb_pixel_total : 176637 time to create 1 rle with new method : 0.004579067230224609 length of segment : 509 time for calcul the mask position with numpy : 0.0012269020080566406 nb_pixel_total : 89544 time to create 1 rle with old method : 0.09639763832092285 length of segment : 614 time for calcul the mask position with numpy : 0.00022840499877929688 nb_pixel_total : 11200 time to create 1 rle with old method : 0.012823820114135742 length of segment : 146 time for calcul the mask position with numpy : 0.00017762184143066406 nb_pixel_total : 11758 time to create 1 rle with old method : 0.01328134536743164 length of segment : 143 time for calcul the mask position with numpy : 0.00021648406982421875 nb_pixel_total : 2563 time to create 1 rle with old method : 0.0030570030212402344 length of segment : 95 time for calcul the mask position with numpy : 0.008950948715209961 nb_pixel_total : 482703 time to create 1 rle with new method : 0.01706218719482422 length of segment : 1301 time for calcul the mask position with numpy : 0.0008442401885986328 nb_pixel_total : 42864 time to create 1 rle with old method : 0.04856705665588379 length of segment : 316 time for calcul the mask position with numpy : 0.0014309883117675781 nb_pixel_total : 117680 time to create 1 rle with old method : 0.17566633224487305 length of segment : 456 time for calcul the mask position with numpy : 0.0012969970703125 nb_pixel_total : 118366 time to create 1 rle with old method : 0.13214373588562012 length of segment : 384 time for calcul the mask position with numpy : 0.003936052322387695 nb_pixel_total : 169817 time to create 1 rle with new method : 0.0073909759521484375 length of segment : 624 time for calcul the mask position with numpy : 0.01651620864868164 nb_pixel_total : 832718 time to create 1 rle with new method : 0.04014873504638672 length of segment : 1552 time for calcul the mask position with numpy : 0.0009608268737792969 nb_pixel_total : 67218 time to create 1 rle with old method : 0.07463812828063965 length of segment : 354 time for calcul the mask position with numpy : 0.0020265579223632812 nb_pixel_total : 139682 time to create 1 rle with old method : 0.158583402633667 length of segment : 588 time for calcul the mask position with numpy : 0.0003120899200439453 nb_pixel_total : 8383 time to create 1 rle with old method : 0.010192632675170898 length of segment : 122 time for calcul the mask position with numpy : 0.002624988555908203 nb_pixel_total : 94900 time to create 1 rle with old method : 0.10692834854125977 length of segment : 394 time for calcul the mask position with numpy : 0.0034666061401367188 nb_pixel_total : 139243 time to create 1 rle with old method : 0.1572413444519043 length of segment : 494 time for calcul the mask position with numpy : 0.006380319595336914 nb_pixel_total : 385402 time to create 1 rle with new method : 0.014077425003051758 length of segment : 1509 time for calcul the mask position with numpy : 0.0006620883941650391 nb_pixel_total : 29839 time to create 1 rle with old method : 0.03399395942687988 length of segment : 231 time for calcul the mask position with numpy : 0.01836872100830078 nb_pixel_total : 1163303 time to create 1 rle with new method : 0.04114532470703125 length of segment : 2140 time for calcul the mask position with numpy : 0.0037746429443359375 nb_pixel_total : 282193 time to create 1 rle with new method : 0.01031184196472168 length of segment : 1002 time for calcul the mask position with numpy : 0.0014946460723876953 nb_pixel_total : 104979 time to create 1 rle with old method : 0.11613917350769043 length of segment : 576 time for calcul the mask position with numpy : 0.00012636184692382812 nb_pixel_total : 2735 time to create 1 rle with old method : 0.003368377685546875 length of segment : 58 time spent for convertir_results : 9.302361488342285 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 69 chid ids of type : 3663 Number RLEs to save : 40585 save missing photos in datou_result : time spend for datou_step_exec : 32.646408796310425 time spend to save output : 2.3719065189361572 total time spend for step 1 : 35.01831531524658 step2:crop_condition Thu Feb 13 08:27:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 69 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1337298060, 1337298032, 1337297908, 1337297906, 1337297876, 1337297873, 1337297804, 1337297802, 1337297768, 1337297722] Looping around the photos to save general results len do output : 32 /-3672974120Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974127Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974126Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974124Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974137Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974090Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974095Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974104Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974119Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974122Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974121Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974078Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974081Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974087Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974088Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974099Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974114Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974128Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974134Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974075Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974086Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974133Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974079Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974080Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974073Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974089Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974082Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974102Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974123Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974131Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974132Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3672974141Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298060', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298032', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297908', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297906', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297876', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297873', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297804', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297802', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297768', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297722', None, None, None, None, None, '2581919') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 106 time used for this insertion : 0.025389909744262695 save_final save missing photos in datou_result : time spend for datou_step_exec : 9.277398347854614 time spend to save output : 0.027108430862426758 total time spend for step 2 : 9.304506778717041 step3:thcl Thu Feb 13 08:27:15 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.0045053958892822266 time to convert the images to numpy array : 3.5762786865234375e-06 time to import caffe and check if the image exist : 0.005038738250732422 time to convert the images to numpy array : 2.86102294921875e-06 time to import caffe and check if the image exist : 0.010563135147094727 time to convert the images to numpy array : 0.03591299057006836 time to import caffe and check if the image exist : 0.006800413131713867 time to convert the images to numpy array : 0.0400543212890625 time to import caffe and check if the image exist : 0.00690770149230957 time to convert the images to numpy array : 0.04474472999572754 time to import caffe and check if the image exist : 0.00908970832824707 time to convert the images to numpy array : 0.04400777816772461 time to import caffe and check if the image exist : 0.004831790924072266 time to convert the images to numpy array : 0.0563197135925293 time to import caffe and check if the image exist : 0.006528615951538086 time to convert the images to numpy array : 0.054414987564086914 time to import caffe and check if the image exist : 0.009202241897583008 time to convert the images to numpy array : 0.05213022232055664 time to import caffe and check if the image exist : 0.008812665939331055 time to convert the images to numpy array : 0.053993940353393555 total time to convert the images to numpy array : 0.29618000984191895 list photo_ids error: [] list photo_ids correct : [-3672974086, -3672974133, -3672974079, -3672974080, -3672974137, -3672974090, -3672974095, -3672974104, -3672974081, -3672974087, -3672974088, -3672974099, -3672974119, -3672974122, -3672974121, -3672974078, -3672974120, -3672974127, -3672974126, -3672974124, -3672974114, -3672974128, -3672974134, -3672974075, -3672974073, -3672974089, -3672974082, -3672974102, -3672974123, -3672974131, -3672974132, -3672974141] number of photos to traite : 32 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10553 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10334 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.2710881233215332 time used to do the prediction : 0.13634705543518066 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10332 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10332 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.2734231948852539 time used to do the prediction : 0.40631890296936035 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.049041748046875e-05 save missing photos in datou_result : time spend for datou_step_exec : 8.265358686447144 time spend to save output : 0.00029206275939941406 total time spend for step 3 : 8.265650749206543 step4:merge_mask_thcl_custom Thu Feb 13 08:27:23 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 69 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1337298060, 1337298032, 1337297908, 1337297906, 1337297876, 1337297873, 1337297804, 1337297802, 1337297768, 1337297722] Looping around the photos to save general results len do output : 10 /1337298060Didn't retrieve data .Didn't retrieve data . /1337298032Didn't retrieve data .Didn't retrieve data . /1337297908Didn't retrieve data .Didn't retrieve data . /1337297906Didn't retrieve data .Didn't retrieve data . /1337297876Didn't retrieve data .Didn't retrieve data . /1337297873Didn't retrieve data .Didn't retrieve data . /1337297804Didn't retrieve data .Didn't retrieve data . /1337297802Didn't retrieve data .Didn't retrieve data . /1337297768Didn't retrieve data .Didn't retrieve data . /1337297722Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298060', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298032', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297908', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297906', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297876', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297873', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297804', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297802', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297768', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297722', None, None, None, None, None, '2581919') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.012950658798217773 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.04993128776550293 time spend to save output : 0.01343536376953125 total time spend for step 4 : 0.06336665153503418 step5:rle_unique_nms_with_priority Thu Feb 13 08:27:23 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 9 nb_hashtags : 6 time to prepare the origin masks : 1.8538472652435303 time for calcul the mask position with numpy : 0.048821449279785156 nb_pixel_total : 1324444 time to create 1 rle with new method : 0.08821678161621094 time for calcul the mask position with numpy : 0.0065860748291015625 nb_pixel_total : 11264 time to create 1 rle with old method : 0.012244701385498047 time for calcul the mask position with numpy : 0.007751941680908203 nb_pixel_total : 3550 time to create 1 rle with old method : 0.003971576690673828 time for calcul the mask position with numpy : 0.00672459602355957 nb_pixel_total : 920 time to create 1 rle with old method : 0.0011830329895019531 time for calcul the mask position with numpy : 0.007166385650634766 nb_pixel_total : 42475 time to create 1 rle with old method : 0.0466310977935791 time for calcul the mask position with numpy : 0.010349273681640625 nb_pixel_total : 559773 time to create 1 rle with new method : 0.20251178741455078 time for calcul the mask position with numpy : 0.0076122283935546875 nb_pixel_total : 5551 time to create 1 rle with old method : 0.008401632308959961 time for calcul the mask position with numpy : 0.007919073104858398 nb_pixel_total : 74160 time to create 1 rle with old method : 0.11509370803833008 time for calcul the mask position with numpy : 0.0067615509033203125 nb_pixel_total : 7223 time to create 1 rle with old method : 0.008530855178833008 time for calcul the mask position with numpy : 0.007054567337036133 nb_pixel_total : 44240 time to create 1 rle with old method : 0.052542686462402344 create new chi : 0.674403190612793 time to delete rle : 0.018535852432250977 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 6741 TO DO : save crop sub photo not yet done ! save time : 0.39286303520202637 nb_obj : 8 nb_hashtags : 5 time to prepare the origin masks : 1.045311450958252 time for calcul the mask position with numpy : 0.042960405349731445 nb_pixel_total : 1380330 time to create 1 rle with new method : 0.2269289493560791 time for calcul the mask position with numpy : 0.00877833366394043 nb_pixel_total : 4844 time to create 1 rle with old method : 0.00707554817199707 time for calcul the mask position with numpy : 0.011123418807983398 nb_pixel_total : 39143 time to create 1 rle with old method : 0.04576921463012695 time for calcul the mask position with numpy : 0.007082939147949219 nb_pixel_total : 85444 time to create 1 rle with old method : 0.12565255165100098 time for calcul the mask position with numpy : 0.007888555526733398 nb_pixel_total : 47700 time to create 1 rle with old method : 0.059416770935058594 time for calcul the mask position with numpy : 0.0075604915618896484 nb_pixel_total : 3918 time to create 1 rle with old method : 0.004978179931640625 time for calcul the mask position with numpy : 0.010752677917480469 nb_pixel_total : 499712 time to create 1 rle with new method : 0.032613515853881836 time for calcul the mask position with numpy : 0.007265329360961914 nb_pixel_total : 5434 time to create 1 rle with old method : 0.006827592849731445 time for calcul the mask position with numpy : 0.007552623748779297 nb_pixel_total : 7075 time to create 1 rle with old method : 0.009192228317260742 create new chi : 0.6335186958312988 time to delete rle : 0.0007510185241699219 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 6136 TO DO : save crop sub photo not yet done ! save time : 0.35892534255981445 nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 0.6535243988037109 time for calcul the mask position with numpy : 0.011647939682006836 nb_pixel_total : 675206 time to create 1 rle with new method : 0.1485917568206787 time for calcul the mask position with numpy : 0.006398677825927734 nb_pixel_total : 26566 time to create 1 rle with old method : 0.029723644256591797 time for calcul the mask position with numpy : 0.006440401077270508 nb_pixel_total : 50200 time to create 1 rle with old method : 0.05540037155151367 time for calcul the mask position with numpy : 0.006673574447631836 nb_pixel_total : 49687 time to create 1 rle with old method : 0.0562748908996582 time for calcul the mask position with numpy : 0.00930476188659668 nb_pixel_total : 122820 time to create 1 rle with old method : 0.13494157791137695 time for calcul the mask position with numpy : 0.007009983062744141 nb_pixel_total : 55108 time to create 1 rle with old method : 0.059987783432006836 time for calcul the mask position with numpy : 0.03722405433654785 nb_pixel_total : 1082143 time to create 1 rle with new method : 0.12208890914916992 time for calcul the mask position with numpy : 0.006123781204223633 nb_pixel_total : 11870 time to create 1 rle with old method : 0.012905359268188477 create new chi : 0.7225408554077148 time to delete rle : 0.0008375644683837891 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 9734 TO DO : save crop sub photo not yet done ! save time : 0.5663723945617676 nb_obj : 9 nb_hashtags : 3 time to prepare the origin masks : 0.7664992809295654 time for calcul the mask position with numpy : 0.028678178787231445 nb_pixel_total : 836562 time to create 1 rle with new method : 0.1107029914855957 time for calcul the mask position with numpy : 0.006670713424682617 nb_pixel_total : 24696 time to create 1 rle with old method : 0.028670549392700195 time for calcul the mask position with numpy : 0.005989789962768555 nb_pixel_total : 833 time to create 1 rle with old method : 0.0010113716125488281 time for calcul the mask position with numpy : 0.006141185760498047 nb_pixel_total : 59831 time to create 1 rle with old method : 0.06482219696044922 time for calcul the mask position with numpy : 0.006821155548095703 nb_pixel_total : 129738 time to create 1 rle with old method : 0.1419374942779541 time for calcul the mask position with numpy : 0.006360530853271484 nb_pixel_total : 1539 time to create 1 rle with old method : 0.001878499984741211 time for calcul the mask position with numpy : 0.006416797637939453 nb_pixel_total : 37666 time to create 1 rle with old method : 0.04904913902282715 time for calcul the mask position with numpy : 0.006352424621582031 nb_pixel_total : 4474 time to create 1 rle with old method : 0.005036354064941406 time for calcul the mask position with numpy : 0.006104946136474609 nb_pixel_total : 18205 time to create 1 rle with old method : 0.01995396614074707 time for calcul the mask position with numpy : 0.012136459350585938 nb_pixel_total : 960056 time to create 1 rle with new method : 0.11722779273986816 create new chi : 0.6485531330108643 time to delete rle : 0.0006372928619384766 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 7145 TO DO : save crop sub photo not yet done ! save time : 0.4128603935241699 nb_obj : 5 nb_hashtags : 1 time to prepare the origin masks : 0.3646996021270752 time for calcul the mask position with numpy : 0.011008501052856445 nb_pixel_total : 732106 time to create 1 rle with new method : 0.030591487884521484 time for calcul the mask position with numpy : 0.007045269012451172 nb_pixel_total : 181862 time to create 1 rle with new method : 0.07450032234191895 time for calcul the mask position with numpy : 0.005865573883056641 nb_pixel_total : 6703 time to create 1 rle with old method : 0.008034467697143555 time for calcul the mask position with numpy : 0.006665945053100586 nb_pixel_total : 141212 time to create 1 rle with old method : 0.1486809253692627 time for calcul the mask position with numpy : 0.006283760070800781 nb_pixel_total : 42719 time to create 1 rle with old method : 0.044637203216552734 time for calcul the mask position with numpy : 0.012167692184448242 nb_pixel_total : 968998 time to create 1 rle with new method : 0.14718961715698242 create new chi : 0.5245568752288818 time to delete rle : 0.0006961822509765625 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 8367 TO DO : save crop sub photo not yet done ! save time : 0.4884018898010254 nb_obj : 5 nb_hashtags : 2 time to prepare the origin masks : 0.5694172382354736 time for calcul the mask position with numpy : 0.08563590049743652 nb_pixel_total : 1185303 time to create 1 rle with new method : 0.1045835018157959 time for calcul the mask position with numpy : 0.008343935012817383 nb_pixel_total : 142527 time to create 1 rle with old method : 0.15452313423156738 time for calcul the mask position with numpy : 0.006402492523193359 nb_pixel_total : 35231 time to create 1 rle with old method : 0.04001164436340332 time for calcul the mask position with numpy : 0.0066013336181640625 nb_pixel_total : 17488 time to create 1 rle with old method : 0.019186735153198242 time for calcul the mask position with numpy : 0.006297111511230469 nb_pixel_total : 5372 time to create 1 rle with old method : 0.00643610954284668 time for calcul the mask position with numpy : 0.012005090713500977 nb_pixel_total : 687679 time to create 1 rle with new method : 0.0841372013092041 create new chi : 0.549548864364624 time to delete rle : 0.0006349086761474609 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6262 TO DO : save crop sub photo not yet done ! save time : 0.3769569396972656 nb_obj : 8 nb_hashtags : 5 time to prepare the origin masks : 0.9226832389831543 time for calcul the mask position with numpy : 0.03537273406982422 nb_pixel_total : 1053406 time to create 1 rle with new method : 0.14818978309631348 time for calcul the mask position with numpy : 0.006333827972412109 nb_pixel_total : 73934 time to create 1 rle with old method : 0.0778799057006836 time for calcul the mask position with numpy : 0.006206512451171875 nb_pixel_total : 561 time to create 1 rle with old method : 0.0006594657897949219 time for calcul the mask position with numpy : 0.006319999694824219 nb_pixel_total : 1525 time to create 1 rle with old method : 0.001737833023071289 time for calcul the mask position with numpy : 0.007211923599243164 nb_pixel_total : 178330 time to create 1 rle with new method : 0.07293915748596191 time for calcul the mask position with numpy : 0.006615877151489258 nb_pixel_total : 89401 time to create 1 rle with old method : 0.09618139266967773 time for calcul the mask position with numpy : 0.010213375091552734 nb_pixel_total : 610525 time to create 1 rle with new method : 0.2326185703277588 time for calcul the mask position with numpy : 0.006567478179931641 nb_pixel_total : 4584 time to create 1 rle with old method : 0.005319833755493164 time for calcul the mask position with numpy : 0.006269216537475586 nb_pixel_total : 61334 time to create 1 rle with old method : 0.06236147880554199 create new chi : 0.8072700500488281 time to delete rle : 0.0006763935089111328 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 7304 TO DO : save crop sub photo not yet done ! save time : 0.4375159740447998 nb_obj : 5 nb_hashtags : 2 time to prepare the origin masks : 0.5966880321502686 time for calcul the mask position with numpy : 0.015271663665771484 nb_pixel_total : 1397388 time to create 1 rle with new method : 0.03074169158935547 time for calcul the mask position with numpy : 0.006182432174682617 nb_pixel_total : 2551 time to create 1 rle with old method : 0.0029871463775634766 time for calcul the mask position with numpy : 0.006322145462036133 nb_pixel_total : 42746 time to create 1 rle with old method : 0.050902605056762695 time for calcul the mask position with numpy : 0.011154890060424805 nb_pixel_total : 505723 time to create 1 rle with new method : 0.031290531158447266 time for calcul the mask position with numpy : 0.0066013336181640625 nb_pixel_total : 7688 time to create 1 rle with old method : 0.012694358825683594 time for calcul the mask position with numpy : 0.007376909255981445 nb_pixel_total : 117504 time to create 1 rle with old method : 0.13370299339294434 create new chi : 0.31619787216186523 time to delete rle : 0.0006477832794189453 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 5443 TO DO : save crop sub photo not yet done ! save time : 0.3205711841583252 nb_obj : 8 nb_hashtags : 4 time to prepare the origin masks : 0.762300968170166 time for calcul the mask position with numpy : 0.012526273727416992 nb_pixel_total : 858578 time to create 1 rle with new method : 0.17701220512390137 time for calcul the mask position with numpy : 0.0066070556640625 nb_pixel_total : 94582 time to create 1 rle with old method : 0.09781455993652344 time for calcul the mask position with numpy : 0.00634312629699707 nb_pixel_total : 115348 time to create 1 rle with old method : 0.12423443794250488 time for calcul the mask position with numpy : 0.006600379943847656 nb_pixel_total : 85546 time to create 1 rle with old method : 0.09144186973571777 time for calcul the mask position with numpy : 0.006181240081787109 nb_pixel_total : 7456 time to create 1 rle with old method : 0.008844137191772461 time for calcul the mask position with numpy : 0.006558656692504883 nb_pixel_total : 67050 time to create 1 rle with old method : 0.07482266426086426 time for calcul the mask position with numpy : 0.00639796257019043 nb_pixel_total : 8333 time to create 1 rle with old method : 0.010204315185546875 time for calcul the mask position with numpy : 0.011565208435058594 nb_pixel_total : 667201 time to create 1 rle with new method : 0.16226553916931152 time for calcul the mask position with numpy : 0.007109642028808594 nb_pixel_total : 169506 time to create 1 rle with new method : 0.030341386795043945 create new chi : 0.8546204566955566 time to delete rle : 0.0007567405700683594 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 8775 TO DO : save crop sub photo not yet done ! save time : 0.4960908889770508 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 0.4129183292388916 time for calcul the mask position with numpy : 0.011116504669189453 nb_pixel_total : 822616 time to create 1 rle with new method : 0.02946186065673828 time for calcul the mask position with numpy : 0.005869865417480469 nb_pixel_total : 1537 time to create 1 rle with old method : 0.0016922950744628906 time for calcul the mask position with numpy : 0.0057108402252197266 nb_pixel_total : 24535 time to create 1 rle with old method : 0.027088642120361328 time for calcul the mask position with numpy : 0.006000041961669922 nb_pixel_total : 23252 time to create 1 rle with old method : 0.026256322860717773 time for calcul the mask position with numpy : 0.013995170593261719 nb_pixel_total : 1171899 time to create 1 rle with new method : 0.07506799697875977 time for calcul the mask position with numpy : 0.0064814090728759766 nb_pixel_total : 29761 time to create 1 rle with old method : 0.03251457214355469 create new chi : 0.2472364902496338 time to delete rle : 0.0006546974182128906 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6985 TO DO : save crop sub photo not yet done ! save time : 0.4019663333892822 map_output_result : {1337298060: (0.0, 'Should be the crop_list due to order', 0.0), 1337298032: (0.0, 'Should be the crop_list due to order', 0.0), 1337297908: (0.0, 'Should be the crop_list due to order', 0.0), 1337297906: (0.0, 'Should be the crop_list due to order', 0.0), 1337297876: (0.0, 'Should be the crop_list due to order', 0.0), 1337297873: (0.0, 'Should be the crop_list due to order', 0.0), 1337297804: (0.0, 'Should be the crop_list due to order', 0.0), 1337297802: (0.0, 'Should be the crop_list due to order', 0.0), 1337297768: (0.0, 'Should be the crop_list due to order', 0.0), 1337297722: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1337298060, 1337298032, 1337297908, 1337297906, 1337297876, 1337297873, 1337297804, 1337297802, 1337297768, 1337297722] Looping around the photos to save general results len do output : 10 /1337298060.Didn't retrieve data . /1337298032.Didn't retrieve data . /1337297908.Didn't retrieve data . /1337297906.Didn't retrieve data . /1337297876.Didn't retrieve data . /1337297873.Didn't retrieve data . /1337297804.Didn't retrieve data . /1337297802.Didn't retrieve data . /1337297768.Didn't retrieve data . /1337297722.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298060', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298032', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297908', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297906', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297876', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297873', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297804', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297802', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297768', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297722', None, None, None, None, None, '2581919') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.012816190719604492 save_final save missing photos in datou_result : time spend for datou_step_exec : 18.510664701461792 time spend to save output : 0.013153076171875 total time spend for step 5 : 18.523817777633667 step6:crop_condition Thu Feb 13 08:27:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 79 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739431663_2865094 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.777780294418335 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739431664_2865094 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8149929046630859 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739431666_2865094 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.1658358573913574 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1337298060, 1337298032, 1337297908, 1337297906, 1337297876, 1337297873, 1337297804, 1337297802, 1337297768, 1337297722] Looping around the photos to save general results len do output : 8 /1337299941Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337299942Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337299945Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337299946Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337299947Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337299948Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337299949Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337299950Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298060', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298032', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297908', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297906', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297876', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297873', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297804', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297802', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297768', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297722', None, None, None, None, None, '2581919') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 34 time used for this insertion : 0.024901390075683594 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.065544605255127 time spend to save output : 0.0254514217376709 total time spend for step 6 : 6.090996026992798 step7:ventilate_hashtags_in_portfolio Thu Feb 13 08:27:48 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20503166 get user id for portfolio 20503166 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20503166 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','autre_refus','Carton_gris','environnement','cartonnette','metal','papier','mal_croppe','Teint_Dans_La_Masse','flou','plastique','Carton_brun')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20503166 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','autre_refus','Carton_gris','environnement','cartonnette','metal','papier','mal_croppe','Teint_Dans_La_Masse','flou','plastique','Carton_brun')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20503166 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','autre_refus','Carton_gris','environnement','cartonnette','metal','papier','mal_croppe','Teint_Dans_La_Masse','flou','plastique','Carton_brun')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20503228,20503229,20503230,20503231,20503232,20503233,20503234,20503235,20503236,20503237,20503238,20503239?tags=kraft,autre_refus,Carton_gris,environnement,cartonnette,metal,papier,mal_croppe,Teint_Dans_La_Masse,flou,plastique,Carton_brun Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1337298060, 1337298032, 1337297908, 1337297906, 1337297876, 1337297873, 1337297804, 1337297802, 1337297768, 1337297722] Looping around the photos to save general results len do output : 1 /20503166. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298060', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298032', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297908', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297906', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297876', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297873', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297804', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297802', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297768', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297722', None, None, None, None, None, '2581919') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013543367385864258 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.5997369289398193 time spend to save output : 0.013814210891723633 total time spend for step 7 : 1.613551139831543 step8:final Thu Feb 13 08:27:49 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1337298060: ('0.12893109513533046',), 1337298032: ('0.12893109513533046',), 1337297908: ('0.12893109513533046',), 1337297906: ('0.12893109513533046',), 1337297876: ('0.12893109513533046',), 1337297873: ('0.12893109513533046',), 1337297804: ('0.12893109513533046',), 1337297802: ('0.12893109513533046',), 1337297768: ('0.12893109513533046',), 1337297722: ('0.12893109513533046',)} new output for save of step final : {1337298060: ('0.12893109513533046',), 1337298032: ('0.12893109513533046',), 1337297908: ('0.12893109513533046',), 1337297906: ('0.12893109513533046',), 1337297876: ('0.12893109513533046',), 1337297873: ('0.12893109513533046',), 1337297804: ('0.12893109513533046',), 1337297802: ('0.12893109513533046',), 1337297768: ('0.12893109513533046',), 1337297722: ('0.12893109513533046',)} [1337298060, 1337298032, 1337297908, 1337297906, 1337297876, 1337297873, 1337297804, 1337297802, 1337297768, 1337297722] Looping around the photos to save general results len do output : 10 /1337298060.Didn't retrieve data . /1337298032.Didn't retrieve data . /1337297908.Didn't retrieve data . /1337297906.Didn't retrieve data . /1337297876.Didn't retrieve data . /1337297873.Didn't retrieve data . /1337297804.Didn't retrieve data . /1337297802.Didn't retrieve data . /1337297768.Didn't retrieve data . /1337297722.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298060', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298032', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297908', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297906', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297876', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297873', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297804', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297802', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297768', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297722', None, None, None, None, None, '2581919') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.0135955810546875 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.09747624397277832 time spend to save output : 0.014136552810668945 total time spend for step 8 : 0.11161279678344727 step9:velours_tree Thu Feb 13 08:27:49 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.03007793426513672 time spend to save output : 3.7670135498046875e-05 total time spend for step 9 : 0.030115604400634766 step10:send_mail_cod Thu Feb 13 08:27:49 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20503166_13-02-2025_08_27_49.pdf 20503228 imagette205032281739431669 20503229 change filename to text .change filename to text .imagette205032291739431669 20503232 imagette205032321739431670 20503233 imagette205032331739431670 20503235 imagette205032351739431670 20503237 imagette205032371739431670 20503238 change filename to text .change filename to text .change filename to text .change filename to text .imagette205032381739431670 20503230 imagette205032301739431670 20503236 change filename to text .change filename to text .imagette205032361739431670 20503239 imagette205032391739431670 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20503166 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20503228,20503229,20503230,20503231,20503232,20503233,20503234,20503235,20503236,20503237,20503238,20503239?tags=kraft,autre_refus,Carton_gris,environnement,cartonnette,metal,papier,mal_croppe,Teint_Dans_La_Masse,flou,plastique,Carton_brun your option no_mail is active, we will not send the real mail to your client args[1337298060] : ('0.12893109513533046',) no score found for photo 1337298060 apple ('0.12893109513533046',) We are sending mail with results at report@fotonower.com args[1337298032] : ('0.12893109513533046',) no score found for photo 1337298032 apple ('0.12893109513533046',) We are sending mail with results at report@fotonower.com args[1337297908] : ('0.12893109513533046',) no score found for photo 1337297908 apple ('0.12893109513533046',) We are sending mail with results at report@fotonower.com args[1337297906] : ('0.12893109513533046',) no score found for photo 1337297906 apple ('0.12893109513533046',) We are sending mail with results at report@fotonower.com args[1337297876] : ('0.12893109513533046',) no score found for photo 1337297876 apple ('0.12893109513533046',) We are sending mail with results at report@fotonower.com args[1337297873] : ('0.12893109513533046',) no score found for photo 1337297873 apple ('0.12893109513533046',) We are sending mail with results at report@fotonower.com args[1337297804] : ('0.12893109513533046',) no score found for photo 1337297804 apple ('0.12893109513533046',) We are sending mail with results at report@fotonower.com args[1337297802] : ('0.12893109513533046',) no score found for photo 1337297802 apple ('0.12893109513533046',) We are sending mail with results at report@fotonower.com args[1337297768] : ('0.12893109513533046',) no score found for photo 1337297768 apple ('0.12893109513533046',) We are sending mail with results at report@fotonower.com args[1337297722] : ('0.12893109513533046',) no score found for photo 1337297722 apple ('0.12893109513533046',) We are sending mail with results at report@fotonower.com refus_total : 0.12893109513533046 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20503166 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1337297720,1337296446,1337296495,1337296499,1337296526,1337296530,1337296562,1337296564,1337297333,1337297336,1337297638,1337297641,1337297683,1337297686,1337297722) Found this number of photos: 15 begin to download photo : 1337297720 begin to download photo : 1337296526 begin to download photo : 1337297333 begin to download photo : 1337297683 download finish for photo 1337296526 begin to download photo : 1337296530 download finish for photo 1337297333 begin to download photo : 1337297336 download finish for photo 1337297720 begin to download photo : 1337296446 download finish for photo 1337297683 begin to download photo : 1337297686 download finish for photo 1337296530 begin to download photo : 1337296562 download finish for photo 1337296446 begin to download photo : 1337296495 download finish for photo 1337297686 begin to download photo : 1337297722 download finish for photo 1337297336 begin to download photo : 1337297638 download finish for photo 1337296495 begin to download photo : 1337296499 download finish for photo 1337296562 begin to download photo : 1337296564 download finish for photo 1337297722 download finish for photo 1337297638 begin to download photo : 1337297641 download finish for photo 1337296499 download finish for photo 1337296564 download finish for photo 1337297641 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20503166_13-02-2025_08_27_49.pdf results_Auto_P20503166_13-02-2025_08_27_49.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20503166_13-02-2025_08_27_49.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20503166','results_Auto_P20503166_13-02-2025_08_27_49.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20503166_13-02-2025_08_27_49.pdf','pdf','','0.19','0.12893109513533046') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1337298060, 1337298032, 1337297908, 1337297906, 1337297876, 1337297873, 1337297804, 1337297802, 1337297768, 1337297722] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298060', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298032', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297908', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297906', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297876', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297873', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297804', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297802', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297768', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297722', None, None, None, None, None, '2581919') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.013806343078613281 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.117384910583496 time spend to save output : 0.01396632194519043 total time spend for step 10 : 2.1313512325286865 step11:split_time_score Thu Feb 13 08:27:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('08', 24),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 13022025 20503166 Nombre de photos uploadées : 24 / 23040 (0%) 13022025 20503166 Nombre de photos taguées (types de déchets): 0 / 24 (0%) 13022025 20503166 Nombre de photos taguées (volume) : 0 / 24 (0%) elapsed_time : load_data_split_time_score 1.1920928955078125e-06 elapsed_time : order_list_meta_photo_and_scores 5.7220458984375e-06 ???????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0012905597686767578 elapsed_time : insert_dashboard_record_day_entry 0.023396968841552734 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.02615516519411045 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20502302_13-02-2025_08_25_34.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20502302 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20502302 AND mptpi.`type`=3726 To do Qualite : 0.00184448534479596 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20502303_13-02-2025_08_21_03.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20502303 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20502303 AND mptpi.`type`=3726 To do Qualite : 0.055052910690238134 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20502304_13-02-2025_08_15_17.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20502304 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20502304 AND mptpi.`type`=3726 To do Qualite : 0.06205589650858348 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20502305_13-02-2025_08_11_14.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20502305 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20502305 AND mptpi.`type`=3726 To do Qualite : 0.12893109513533046 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20503166_13-02-2025_08_27_49.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20503166 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20503166 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'13022025': {'nb_upload': 24, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1337298060, 1337298032, 1337297908, 1337297906, 1337297876, 1337297873, 1337297804, 1337297802, 1337297768, 1337297722] Looping around the photos to save general results len do output : 1 /20503166Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298060', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337298032', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297908', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297906', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297876', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297873', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297804', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297802', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297768', None, None, None, None, None, '2581919') ('3459', None, None, None, None, None, None, None, '2581919') ('3459', '20503166', '1337297722', None, None, None, None, None, '2581919') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013226032257080078 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.46732521057128906 time spend to save output : 0.013353824615478516 total time spend for step 11 : 0.4806790351867676 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 51.08user 25.73system 1:26.54elapsed 88%CPU (0avgtext+0avgdata 2775880maxresident)k 68568inputs+33240outputs (451major+1904470minor)pagefaults 0swaps