python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 287486 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 17>10 please execute split_portfolio.py -i 21131115 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2646484'] with mtr_portfolio_ids : ['21131115'] and first list_photo_ids : [] new path : /proc/287486/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 7 ; length of list_pids : 7 ; length of list_args : 7 time to download the photos : 1.3830506801605225 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Mar 7 00:54:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4607 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-07 00:54:36.370164: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-07 00:54:36.403289: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-07 00:54:36.405549: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f5950000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-07 00:54:36.405585: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-07 00:54:36.410331: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-07 00:54:36.674881: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x13e07f90 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-07 00:54:36.674946: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-07 00:54:36.676141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 00:54:36.678735: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 00:54:36.706679: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 00:54:36.728288: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 00:54:36.733235: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 00:54:36.769397: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 00:54:36.774054: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 00:54:36.836385: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 00:54:36.837582: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 00:54:36.838065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 00:54:36.838968: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-07 00:54:36.838986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-07 00:54:36.838996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-07 00:54:36.840599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4155 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-07 00:54:37.311877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 00:54:37.312007: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 00:54:37.312038: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 00:54:37.312065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 00:54:37.312090: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 00:54:37.312114: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 00:54:37.312138: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 00:54:37.312164: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 00:54:37.313337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 00:54:37.314549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 00:54:37.314596: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 00:54:37.314620: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 00:54:37.314641: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 00:54:37.314663: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 00:54:37.314684: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 00:54:37.314704: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 00:54:37.314724: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 00:54:37.315896: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 00:54:37.315940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-07 00:54:37.315952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-07 00:54:37.315962: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-07 00:54:37.317154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4155 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-07 00:54:47.059400: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 00:54:47.404122: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 00:54:49.625735: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.708558: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-03-07 00:54:50.791092: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.791198: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:54:50.793108: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.793147: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:54:50.800572: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.800597: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:54:50.801265: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.801281: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:54:50.873315: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.873344: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:54:50.874012: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.874027: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:54:50.882058: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.882079: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 417.62MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:54:50.882745: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.882759: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 417.62MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:54:50.906476: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.906528: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 188.27MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:54:50.907449: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.907471: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 188.27MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:54:50.908366: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.909096: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.913309: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.914042: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.914789: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.915540: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.917751: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.917767: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-03-07 00:54:50.926995: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.927730: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.936835: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.937581: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.938317: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.939061: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.939801: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:54:50.940529: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.56G (3821928448 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 7 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 13 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 27 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 23 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 10 Detection mask done ! Trying to reset tf kernel 287853 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3414 tf kernel not reseted sub process len(results) : 7 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 7 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4607 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.002557992935180664 nb_pixel_total : 75128 time to create 1 rle with old method : 0.08808779716491699 length of segment : 336 time for calcul the mask position with numpy : 1.20338773727417 nb_pixel_total : 3134006 time to create 1 rle with new method : 0.23522138595581055 length of segment : 3040 time for calcul the mask position with numpy : 0.0013768672943115234 nb_pixel_total : 49705 time to create 1 rle with old method : 0.05607962608337402 length of segment : 350 time for calcul the mask position with numpy : 0.0016031265258789062 nb_pixel_total : 59913 time to create 1 rle with old method : 0.06925272941589355 length of segment : 325 time for calcul the mask position with numpy : 0.0037839412689208984 nb_pixel_total : 185963 time to create 1 rle with new method : 0.009202957153320312 length of segment : 451 time for calcul the mask position with numpy : 0.003482818603515625 nb_pixel_total : 95389 time to create 1 rle with old method : 0.10367250442504883 length of segment : 331 time for calcul the mask position with numpy : 0.15022659301757812 nb_pixel_total : 1492610 time to create 1 rle with new method : 0.25112223625183105 length of segment : 1742 time for calcul the mask position with numpy : 0.0005393028259277344 nb_pixel_total : 15650 time to create 1 rle with old method : 0.018168210983276367 length of segment : 233 time for calcul the mask position with numpy : 0.002138376235961914 nb_pixel_total : 91125 time to create 1 rle with old method : 0.0989680290222168 length of segment : 501 time for calcul the mask position with numpy : 0.0030698776245117188 nb_pixel_total : 162139 time to create 1 rle with new method : 0.0054361820220947266 length of segment : 400 time for calcul the mask position with numpy : 0.0005645751953125 nb_pixel_total : 30581 time to create 1 rle with old method : 0.04386258125305176 length of segment : 207 time for calcul the mask position with numpy : 0.0002961158752441406 nb_pixel_total : 4200 time to create 1 rle with old method : 0.005181312561035156 length of segment : 109 time for calcul the mask position with numpy : 0.0010976791381835938 nb_pixel_total : 35921 time to create 1 rle with old method : 0.04041624069213867 length of segment : 211 time for calcul the mask position with numpy : 0.0001201629638671875 nb_pixel_total : 917 time to create 1 rle with old method : 0.0012106895446777344 length of segment : 28 time for calcul the mask position with numpy : 0.0002243518829345703 nb_pixel_total : 4570 time to create 1 rle with old method : 0.005666494369506836 length of segment : 48 time for calcul the mask position with numpy : 0.0012335777282714844 nb_pixel_total : 43137 time to create 1 rle with old method : 0.04955649375915527 length of segment : 224 time for calcul the mask position with numpy : 0.0003936290740966797 nb_pixel_total : 5335 time to create 1 rle with old method : 0.00879049301147461 length of segment : 140 time for calcul the mask position with numpy : 0.003167390823364258 nb_pixel_total : 126905 time to create 1 rle with old method : 0.14241766929626465 length of segment : 403 time for calcul the mask position with numpy : 0.0007212162017822266 nb_pixel_total : 23714 time to create 1 rle with old method : 0.029165267944335938 length of segment : 118 time for calcul the mask position with numpy : 0.00024080276489257812 nb_pixel_total : 3798 time to create 1 rle with old method : 0.004642963409423828 length of segment : 107 time for calcul the mask position with numpy : 0.0019989013671875 nb_pixel_total : 58387 time to create 1 rle with old method : 0.06572651863098145 length of segment : 310 time for calcul the mask position with numpy : 0.0002529621124267578 nb_pixel_total : 4279 time to create 1 rle with old method : 0.005480766296386719 length of segment : 51 time for calcul the mask position with numpy : 0.0026140213012695312 nb_pixel_total : 58066 time to create 1 rle with old method : 0.06745529174804688 length of segment : 514 time for calcul the mask position with numpy : 0.00015616416931152344 nb_pixel_total : 1435 time to create 1 rle with old method : 0.001912832260131836 length of segment : 29 time for calcul the mask position with numpy : 0.004893302917480469 nb_pixel_total : 183088 time to create 1 rle with new method : 0.007184743881225586 length of segment : 542 time for calcul the mask position with numpy : 0.004888772964477539 nb_pixel_total : 124988 time to create 1 rle with old method : 0.14000272750854492 length of segment : 693 time for calcul the mask position with numpy : 0.0006346702575683594 nb_pixel_total : 7164 time to create 1 rle with old method : 0.012224674224853516 length of segment : 95 time for calcul the mask position with numpy : 0.00018668174743652344 nb_pixel_total : 1978 time to create 1 rle with old method : 0.0034329891204833984 length of segment : 40 time for calcul the mask position with numpy : 0.00036334991455078125 nb_pixel_total : 3282 time to create 1 rle with old method : 0.005738973617553711 length of segment : 86 time for calcul the mask position with numpy : 0.00031876564025878906 nb_pixel_total : 6298 time to create 1 rle with old method : 0.007357120513916016 length of segment : 103 time for calcul the mask position with numpy : 0.0001049041748046875 nb_pixel_total : 1654 time to create 1 rle with old method : 0.002240419387817383 length of segment : 80 time for calcul the mask position with numpy : 0.0012526512145996094 nb_pixel_total : 59163 time to create 1 rle with old method : 0.06897759437561035 length of segment : 519 time for calcul the mask position with numpy : 0.0037736892700195312 nb_pixel_total : 105003 time to create 1 rle with old method : 0.11638164520263672 length of segment : 587 time for calcul the mask position with numpy : 0.0011496543884277344 nb_pixel_total : 51117 time to create 1 rle with old method : 0.057935237884521484 length of segment : 312 time for calcul the mask position with numpy : 0.002061128616333008 nb_pixel_total : 73044 time to create 1 rle with old method : 0.08100461959838867 length of segment : 475 time for calcul the mask position with numpy : 0.0017333030700683594 nb_pixel_total : 73893 time to create 1 rle with old method : 0.08504962921142578 length of segment : 331 time for calcul the mask position with numpy : 0.002098560333251953 nb_pixel_total : 112881 time to create 1 rle with old method : 0.1257648468017578 length of segment : 317 time for calcul the mask position with numpy : 0.0023620128631591797 nb_pixel_total : 104788 time to create 1 rle with old method : 0.11774349212646484 length of segment : 617 time for calcul the mask position with numpy : 0.002966165542602539 nb_pixel_total : 182716 time to create 1 rle with new method : 0.005547761917114258 length of segment : 414 time for calcul the mask position with numpy : 0.0006983280181884766 nb_pixel_total : 25931 time to create 1 rle with old method : 0.029786109924316406 length of segment : 159 time for calcul the mask position with numpy : 0.0004901885986328125 nb_pixel_total : 11749 time to create 1 rle with old method : 0.014792203903198242 length of segment : 321 time for calcul the mask position with numpy : 0.001096963882446289 nb_pixel_total : 34639 time to create 1 rle with old method : 0.04129147529602051 length of segment : 225 time for calcul the mask position with numpy : 0.0008602142333984375 nb_pixel_total : 22825 time to create 1 rle with old method : 0.02677750587463379 length of segment : 288 time for calcul the mask position with numpy : 0.005808830261230469 nb_pixel_total : 191662 time to create 1 rle with new method : 0.012820005416870117 length of segment : 505 time for calcul the mask position with numpy : 0.0002498626708984375 nb_pixel_total : 4590 time to create 1 rle with old method : 0.005524635314941406 length of segment : 74 time for calcul the mask position with numpy : 0.005151271820068359 nb_pixel_total : 206578 time to create 1 rle with new method : 0.008065938949584961 length of segment : 550 time for calcul the mask position with numpy : 0.000202178955078125 nb_pixel_total : 3035 time to create 1 rle with old method : 0.003659486770629883 length of segment : 73 time for calcul the mask position with numpy : 0.003673076629638672 nb_pixel_total : 115350 time to create 1 rle with old method : 0.1338052749633789 length of segment : 549 time for calcul the mask position with numpy : 0.0007424354553222656 nb_pixel_total : 15161 time to create 1 rle with old method : 0.021645784378051758 length of segment : 190 time for calcul the mask position with numpy : 0.01114034652709961 nb_pixel_total : 512485 time to create 1 rle with new method : 0.04100775718688965 length of segment : 986 time for calcul the mask position with numpy : 0.0037543773651123047 nb_pixel_total : 177178 time to create 1 rle with new method : 0.007746696472167969 length of segment : 586 time for calcul the mask position with numpy : 0.0035190582275390625 nb_pixel_total : 130547 time to create 1 rle with old method : 0.1693131923675537 length of segment : 432 time for calcul the mask position with numpy : 0.000621795654296875 nb_pixel_total : 17359 time to create 1 rle with old method : 0.02022075653076172 length of segment : 184 time for calcul the mask position with numpy : 0.0004589557647705078 nb_pixel_total : 16414 time to create 1 rle with old method : 0.019579172134399414 length of segment : 113 time for calcul the mask position with numpy : 0.00023555755615234375 nb_pixel_total : 15752 time to create 1 rle with old method : 0.01830601692199707 length of segment : 175 time for calcul the mask position with numpy : 0.0008525848388671875 nb_pixel_total : 21460 time to create 1 rle with old method : 0.02781081199645996 length of segment : 251 time for calcul the mask position with numpy : 0.0030908584594726562 nb_pixel_total : 120502 time to create 1 rle with old method : 0.14176225662231445 length of segment : 393 time for calcul the mask position with numpy : 0.00017762184143066406 nb_pixel_total : 4259 time to create 1 rle with old method : 0.005237579345703125 length of segment : 116 time for calcul the mask position with numpy : 0.00591278076171875 nb_pixel_total : 68387 time to create 1 rle with old method : 0.07699394226074219 length of segment : 359 time for calcul the mask position with numpy : 0.0023217201232910156 nb_pixel_total : 56276 time to create 1 rle with old method : 0.0627892017364502 length of segment : 439 time for calcul the mask position with numpy : 0.0011818408966064453 nb_pixel_total : 36306 time to create 1 rle with old method : 0.04206395149230957 length of segment : 260 time for calcul the mask position with numpy : 0.0006422996520996094 nb_pixel_total : 7081 time to create 1 rle with old method : 0.008782148361206055 length of segment : 191 time for calcul the mask position with numpy : 0.0017817020416259766 nb_pixel_total : 83086 time to create 1 rle with old method : 0.09214043617248535 length of segment : 289 time for calcul the mask position with numpy : 0.00809478759765625 nb_pixel_total : 244256 time to create 1 rle with new method : 0.01333165168762207 length of segment : 616 time for calcul the mask position with numpy : 0.0020232200622558594 nb_pixel_total : 39168 time to create 1 rle with old method : 0.04618072509765625 length of segment : 305 time for calcul the mask position with numpy : 0.001476287841796875 nb_pixel_total : 33193 time to create 1 rle with old method : 0.03924393653869629 length of segment : 271 time for calcul the mask position with numpy : 0.0016286373138427734 nb_pixel_total : 36742 time to create 1 rle with old method : 0.042891740798950195 length of segment : 345 time for calcul the mask position with numpy : 0.0006532669067382812 nb_pixel_total : 32749 time to create 1 rle with old method : 0.03807234764099121 length of segment : 233 time for calcul the mask position with numpy : 0.00054168701171875 nb_pixel_total : 13626 time to create 1 rle with old method : 0.01636052131652832 length of segment : 134 time for calcul the mask position with numpy : 0.0005426406860351562 nb_pixel_total : 28637 time to create 1 rle with old method : 0.03328371047973633 length of segment : 244 time for calcul the mask position with numpy : 0.0010187625885009766 nb_pixel_total : 22900 time to create 1 rle with old method : 0.03691697120666504 length of segment : 211 time for calcul the mask position with numpy : 0.0006451606750488281 nb_pixel_total : 22760 time to create 1 rle with old method : 0.026464223861694336 length of segment : 153 time for calcul the mask position with numpy : 0.002629995346069336 nb_pixel_total : 81917 time to create 1 rle with old method : 0.09256911277770996 length of segment : 338 time for calcul the mask position with numpy : 0.0002167224884033203 nb_pixel_total : 3130 time to create 1 rle with old method : 0.003857135772705078 length of segment : 102 time for calcul the mask position with numpy : 0.0029511451721191406 nb_pixel_total : 81133 time to create 1 rle with old method : 0.0941314697265625 length of segment : 642 time for calcul the mask position with numpy : 0.0006783008575439453 nb_pixel_total : 19318 time to create 1 rle with old method : 0.03211832046508789 length of segment : 113 time for calcul the mask position with numpy : 0.00019311904907226562 nb_pixel_total : 4173 time to create 1 rle with old method : 0.004868507385253906 length of segment : 89 time for calcul the mask position with numpy : 0.00040268898010253906 nb_pixel_total : 8926 time to create 1 rle with old method : 0.010621070861816406 length of segment : 140 time for calcul the mask position with numpy : 0.0006287097930908203 nb_pixel_total : 12825 time to create 1 rle with old method : 0.015000581741333008 length of segment : 154 time for calcul the mask position with numpy : 0.0003445148468017578 nb_pixel_total : 5964 time to create 1 rle with old method : 0.007064342498779297 length of segment : 119 time for calcul the mask position with numpy : 0.0018124580383300781 nb_pixel_total : 57582 time to create 1 rle with old method : 0.06600618362426758 length of segment : 343 time for calcul the mask position with numpy : 0.00940394401550293 nb_pixel_total : 410163 time to create 1 rle with new method : 0.01269674301147461 length of segment : 900 time for calcul the mask position with numpy : 0.0002760887145996094 nb_pixel_total : 10669 time to create 1 rle with old method : 0.012796640396118164 length of segment : 141 time for calcul the mask position with numpy : 0.0017924308776855469 nb_pixel_total : 74976 time to create 1 rle with old method : 0.08264446258544922 length of segment : 332 time for calcul the mask position with numpy : 0.00024509429931640625 nb_pixel_total : 4383 time to create 1 rle with old method : 0.0055694580078125 length of segment : 144 time for calcul the mask position with numpy : 0.00023627281188964844 nb_pixel_total : 6733 time to create 1 rle with old method : 0.008279561996459961 length of segment : 72 time for calcul the mask position with numpy : 0.0012826919555664062 nb_pixel_total : 34231 time to create 1 rle with old method : 0.04061245918273926 length of segment : 242 time for calcul the mask position with numpy : 0.001409292221069336 nb_pixel_total : 49856 time to create 1 rle with old method : 0.05761122703552246 length of segment : 311 time for calcul the mask position with numpy : 0.00011086463928222656 nb_pixel_total : 1451 time to create 1 rle with old method : 0.0019257068634033203 length of segment : 32 time for calcul the mask position with numpy : 0.00010013580322265625 nb_pixel_total : 1330 time to create 1 rle with old method : 0.0017545223236083984 length of segment : 48 time for calcul the mask position with numpy : 0.0003199577331542969 nb_pixel_total : 8777 time to create 1 rle with old method : 0.010223627090454102 length of segment : 148 time for calcul the mask position with numpy : 0.0032262802124023438 nb_pixel_total : 119884 time to create 1 rle with old method : 0.1449575424194336 length of segment : 345 time for calcul the mask position with numpy : 0.0002930164337158203 nb_pixel_total : 3565 time to create 1 rle with old method : 0.00630497932434082 length of segment : 75 time for calcul the mask position with numpy : 0.0002987384796142578 nb_pixel_total : 4432 time to create 1 rle with old method : 0.009168148040771484 length of segment : 85 time for calcul the mask position with numpy : 0.004191160202026367 nb_pixel_total : 155135 time to create 1 rle with new method : 0.007429599761962891 length of segment : 367 time for calcul the mask position with numpy : 0.00030350685119628906 nb_pixel_total : 3892 time to create 1 rle with old method : 0.007082939147949219 length of segment : 61 time for calcul the mask position with numpy : 0.005954265594482422 nb_pixel_total : 68327 time to create 1 rle with old method : 0.08640480041503906 length of segment : 675 time for calcul the mask position with numpy : 0.0030548572540283203 nb_pixel_total : 126972 time to create 1 rle with old method : 0.16535520553588867 length of segment : 263 time for calcul the mask position with numpy : 0.0028138160705566406 nb_pixel_total : 149911 time to create 1 rle with old method : 0.2328202724456787 length of segment : 369 time for calcul the mask position with numpy : 0.00042510032653808594 nb_pixel_total : 22166 time to create 1 rle with old method : 0.02674126625061035 length of segment : 235 time spent for convertir_results : 12.304213762283325 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 100 chid ids of type : 3663 Number RLEs to save : 32524 save missing photos in datou_result : time spend for datou_step_exec : 57.87419295310974 time spend to save output : 3.9830031394958496 total time spend for step 1 : 61.85719609260559 step2:crop_condition Fri Mar 7 00:55:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 7 ! batch 1 Loaded 100 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 29 About to insert : list_path_to_insert length 29 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1341265967, 1341265754, 1341265698, 1341265629, 1341265574, 1341265562, 1341265555] Looping around the photos to save general results len do output : 61 /-3696632059Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632081Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632087Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632091Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632124Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632128Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632123Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632052Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632062Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632115Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632051Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632050Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632075Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632067Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632073Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632089Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632114Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632105Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632106Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632108Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632140Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632047Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632044Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632049Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632074Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632066Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632064Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632072Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632057Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632080Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632079Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632076Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632085Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632093Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632090Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632092Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632094Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632102Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632107Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632113Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632101Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632104Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632129Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632120Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632122Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632125Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632133Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632138Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632136Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632139Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632042Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632058Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632095Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632119Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632054Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632071Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632083Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632077Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632098Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632116Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696632111Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265967', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265754', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265698', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265629', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265574', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265562', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265555', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 190 time used for this insertion : 0.13706064224243164 save_final save missing photos in datou_result : time spend for datou_step_exec : 29.848695993423462 time spend to save output : 0.13964438438415527 total time spend for step 2 : 29.988340377807617 step3:thcl Fri Mar 7 00:55:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.0076558589935302734 time to convert the images to numpy array : 3.5762786865234375e-06 time to import caffe and check if the image exist : 0.011298894882202148 time to convert the images to numpy array : 0.046871185302734375 time to import caffe and check if the image exist : 0.014756441116333008 time to convert the images to numpy array : 0.06574702262878418 time to import caffe and check if the image exist : 0.01512002944946289 time to convert the images to numpy array : 0.07878732681274414 time to import caffe and check if the image exist : 0.014564037322998047 time to convert the images to numpy array : 0.09391093254089355 time to import caffe and check if the image exist : 0.01645183563232422 time to convert the images to numpy array : 0.09453678131103516 time to import caffe and check if the image exist : 0.015077829360961914 time to convert the images to numpy array : 0.10329270362854004 time to import caffe and check if the image exist : 0.012221813201904297 time to convert the images to numpy array : 0.11452269554138184 time to import caffe and check if the image exist : 0.011027097702026367 time to convert the images to numpy array : 0.4297056198120117 time to import caffe and check if the image exist : 0.008270740509033203 time to convert the images to numpy array : 0.47771596908569336 total time to convert the images to numpy array : 0.8833398818969727 list photo_ids error: [] list photo_ids correct : [-3696632139, -3696632042, -3696632058, -3696632095, -3696632119, -3696632054, -3696632071, -3696632083, -3696632077, -3696632098, -3696632116, -3696632111, -3696632052, -3696632062, -3696632115, -3696632051, -3696632050, -3696632075, -3696632067, -3696632092, -3696632094, -3696632102, -3696632107, -3696632113, -3696632101, -3696632104, -3696632047, -3696632044, -3696632049, -3696632074, -3696632066, -3696632064, -3696632072, -3696632129, -3696632120, -3696632122, -3696632125, -3696632133, -3696632138, -3696632136, -3696632057, -3696632080, -3696632079, -3696632076, -3696632085, -3696632093, -3696632090, -3696632073, -3696632089, -3696632114, -3696632105, -3696632106, -3696632108, -3696632140, -3696632059, -3696632081, -3696632087, -3696632091, -3696632124, -3696632128, -3696632123] number of photos to traite : 61 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4607 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4388 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.731276273727417 time used to do the prediction : 0.2390298843383789 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4607 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4607 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.7966556549072266 time used to do the prediction : 0.13240551948547363 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.8835067749023438e-05 save missing photos in datou_result : time spend for datou_step_exec : 9.817011594772339 time spend to save output : 0.0005023479461669922 total time spend for step 3 : 9.817513942718506 step4:merge_mask_thcl_custom Fri Mar 7 00:56:09 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 100 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1341265967, 1341265754, 1341265698, 1341265629, 1341265574, 1341265562, 1341265555] Looping around the photos to save general results len do output : 7 /1341265967Didn't retrieve data .Didn't retrieve data . /1341265754Didn't retrieve data .Didn't retrieve data . /1341265698Didn't retrieve data .Didn't retrieve data . /1341265629Didn't retrieve data .Didn't retrieve data . /1341265574Didn't retrieve data .Didn't retrieve data . /1341265562Didn't retrieve data .Didn't retrieve data . /1341265555Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265967', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265754', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265698', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265629', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265574', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265562', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265555', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 21 time used for this insertion : 0.014712810516357422 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.060411930084228516 time spend to save output : 0.015112161636352539 total time spend for step 4 : 0.07552409172058105 step5:rle_unique_nms_with_priority Fri Mar 7 00:56:09 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 11 nb_hashtags : 5 time to prepare the origin masks : 4.633201599121094 time for calcul the mask position with numpy : 0.13640975952148438 nb_pixel_total : 2817262 time to create 1 rle with new method : 0.48473668098449707 time for calcul the mask position with numpy : 0.02283191680908203 nb_pixel_total : 49665 time to create 1 rle with old method : 0.07873392105102539 time for calcul the mask position with numpy : 0.020727872848510742 nb_pixel_total : 6950 time to create 1 rle with old method : 0.00800323486328125 time for calcul the mask position with numpy : 0.0224454402923584 nb_pixel_total : 389494 time to create 1 rle with new method : 0.43228936195373535 time for calcul the mask position with numpy : 0.022690296173095703 nb_pixel_total : 185835 time to create 1 rle with new method : 0.5732994079589844 time for calcul the mask position with numpy : 0.02420496940612793 nb_pixel_total : 155587 time to create 1 rle with new method : 0.33180832862854004 time for calcul the mask position with numpy : 0.021876811981201172 nb_pixel_total : 59863 time to create 1 rle with old method : 0.06488513946533203 time for calcul the mask position with numpy : 0.021205663681030273 nb_pixel_total : 72398 time to create 1 rle with old method : 0.07911849021911621 time for calcul the mask position with numpy : 0.021601200103759766 nb_pixel_total : 75547 time to create 1 rle with old method : 0.08125662803649902 time for calcul the mask position with numpy : 0.02087569236755371 nb_pixel_total : 90887 time to create 1 rle with old method : 0.10079646110534668 time for calcul the mask position with numpy : 0.24870944023132324 nb_pixel_total : 3131120 time to create 1 rle with new method : 0.3982846736907959 time for calcul the mask position with numpy : 0.02116703987121582 nb_pixel_total : 15632 time to create 1 rle with old method : 0.01740574836730957 create new chi : 3.389690637588501 time to delete rle : 0.01615452766418457 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 15388 TO DO : save crop sub photo not yet done ! save time : 1.426680088043213 nb_obj : 23 nb_hashtags : 6 time to prepare the origin masks : 23.070984363555908 time for calcul the mask position with numpy : 0.6838560104370117 nb_pixel_total : 6197991 time to create 1 rle with new method : 1.2164978981018066 time for calcul the mask position with numpy : 0.03597831726074219 nb_pixel_total : 182833 time to create 1 rle with new method : 1.306293249130249 time for calcul the mask position with numpy : 0.03450298309326172 nb_pixel_total : 104898 time to create 1 rle with old method : 0.11940407752990723 time for calcul the mask position with numpy : 0.029729604721069336 nb_pixel_total : 3702 time to create 1 rle with old method : 0.004336118698120117 time for calcul the mask position with numpy : 0.03329753875732422 nb_pixel_total : 126713 time to create 1 rle with old method : 0.14111042022705078 time for calcul the mask position with numpy : 0.038201093673706055 nb_pixel_total : 58224 time to create 1 rle with old method : 0.0674285888671875 time for calcul the mask position with numpy : 0.04852914810180664 nb_pixel_total : 35810 time to create 1 rle with old method : 0.04503679275512695 time for calcul the mask position with numpy : 0.044943809509277344 nb_pixel_total : 3269 time to create 1 rle with old method : 0.004120349884033203 time for calcul the mask position with numpy : 0.04265332221984863 nb_pixel_total : 1963 time to create 1 rle with old method : 0.002307415008544922 time for calcul the mask position with numpy : 0.03341221809387207 nb_pixel_total : 7093 time to create 1 rle with old method : 0.007993221282958984 time for calcul the mask position with numpy : 0.03442883491516113 nb_pixel_total : 6284 time to create 1 rle with old method : 0.007021427154541016 time for calcul the mask position with numpy : 0.034468650817871094 nb_pixel_total : 1426 time to create 1 rle with old method : 0.0016624927520751953 time for calcul the mask position with numpy : 0.0351862907409668 nb_pixel_total : 4214 time to create 1 rle with old method : 0.004708766937255859 time for calcul the mask position with numpy : 0.03389549255371094 nb_pixel_total : 3747 time to create 1 rle with old method : 0.003984689712524414 time for calcul the mask position with numpy : 0.03627204895019531 nb_pixel_total : 23588 time to create 1 rle with old method : 0.026427030563354492 time for calcul the mask position with numpy : 0.04003500938415527 nb_pixel_total : 50982 time to create 1 rle with old method : 0.07599687576293945 time for calcul the mask position with numpy : 0.03613686561584473 nb_pixel_total : 4509 time to create 1 rle with old method : 0.005124807357788086 time for calcul the mask position with numpy : 0.0351567268371582 nb_pixel_total : 451 time to create 1 rle with old method : 0.0006067752838134766 time for calcul the mask position with numpy : 0.033574581146240234 nb_pixel_total : 907 time to create 1 rle with old method : 0.0010607242584228516 time for calcul the mask position with numpy : 0.03617596626281738 nb_pixel_total : 124835 time to create 1 rle with old method : 0.13712716102600098 time for calcul the mask position with numpy : 0.03572416305541992 nb_pixel_total : 43036 time to create 1 rle with old method : 0.046334266662597656 time for calcul the mask position with numpy : 0.037140607833862305 nb_pixel_total : 4153 time to create 1 rle with old method : 0.006783485412597656 time for calcul the mask position with numpy : 0.0393373966217041 nb_pixel_total : 1572 time to create 1 rle with old method : 0.003084421157836914 time for calcul the mask position with numpy : 0.03862166404724121 nb_pixel_total : 58040 time to create 1 rle with old method : 0.06518220901489258 create new chi : 4.8915345668792725 time to delete rle : 0.0018820762634277344 batch 1 Loaded 24 chid ids of type : 3726 Number RLEs to save : 12365 TO DO : save crop sub photo not yet done ! save time : 0.7969624996185303 nb_obj : 11 nb_hashtags : 6 time to prepare the origin masks : 14.451607942581177 time for calcul the mask position with numpy : 0.5813546180725098 nb_pixel_total : 6322483 time to create 1 rle with new method : 0.4620175361633301 time for calcul the mask position with numpy : 0.035880327224731445 nb_pixel_total : 73681 time to create 1 rle with old method : 0.07954621315002441 time for calcul the mask position with numpy : 0.03603529930114746 nb_pixel_total : 191181 time to create 1 rle with new method : 0.38779783248901367 time for calcul the mask position with numpy : 0.03557395935058594 nb_pixel_total : 4563 time to create 1 rle with old method : 0.005330801010131836 time for calcul the mask position with numpy : 0.03515481948852539 nb_pixel_total : 75031 time to create 1 rle with old method : 0.08181405067443848 time for calcul the mask position with numpy : 0.02958965301513672 nb_pixel_total : 22799 time to create 1 rle with old method : 0.02688908576965332 time for calcul the mask position with numpy : 0.030863523483276367 nb_pixel_total : 11697 time to create 1 rle with old method : 0.5216445922851562 time for calcul the mask position with numpy : 0.0321652889251709 nb_pixel_total : 69962 time to create 1 rle with old method : 0.08116030693054199 time for calcul the mask position with numpy : 0.02168440818786621 nb_pixel_total : 112797 time to create 1 rle with old method : 0.13750433921813965 time for calcul the mask position with numpy : 0.028189659118652344 nb_pixel_total : 25753 time to create 1 rle with old method : 0.028941869735717773 time for calcul the mask position with numpy : 0.023548126220703125 nb_pixel_total : 105774 time to create 1 rle with old method : 0.1244513988494873 time for calcul the mask position with numpy : 0.02465677261352539 nb_pixel_total : 34519 time to create 1 rle with old method : 0.03858470916748047 create new chi : 2.9558968544006348 time to delete rle : 0.0009200572967529297 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 9166 TO DO : save crop sub photo not yet done ! save time : 0.8467452526092529 nb_obj : 11 nb_hashtags : 6 time to prepare the origin masks : 11.824493169784546 time for calcul the mask position with numpy : 1.0030288696289062 nb_pixel_total : 5728244 time to create 1 rle with new method : 0.7788209915161133 time for calcul the mask position with numpy : 0.03675103187561035 nb_pixel_total : 16350 time to create 1 rle with old method : 0.01728200912475586 time for calcul the mask position with numpy : 0.037940025329589844 nb_pixel_total : 4228 time to create 1 rle with old method : 0.004507541656494141 time for calcul the mask position with numpy : 0.03983569145202637 nb_pixel_total : 15025 time to create 1 rle with old method : 0.01815938949584961 time for calcul the mask position with numpy : 0.044661521911621094 nb_pixel_total : 17350 time to create 1 rle with old method : 0.018494367599487305 time for calcul the mask position with numpy : 0.04697704315185547 nb_pixel_total : 206017 time to create 1 rle with new method : 0.6212141513824463 time for calcul the mask position with numpy : 0.03999471664428711 nb_pixel_total : 120325 time to create 1 rle with old method : 0.15311646461486816 time for calcul the mask position with numpy : 0.05203580856323242 nb_pixel_total : 518203 time to create 1 rle with new method : 0.41174888610839844 time for calcul the mask position with numpy : 0.051851749420166016 nb_pixel_total : 115012 time to create 1 rle with old method : 0.14444875717163086 time for calcul the mask position with numpy : 0.05076956748962402 nb_pixel_total : 129503 time to create 1 rle with old method : 0.14312100410461426 time for calcul the mask position with numpy : 0.039078712463378906 nb_pixel_total : 3017 time to create 1 rle with old method : 0.0033941268920898438 time for calcul the mask position with numpy : 0.02997112274169922 nb_pixel_total : 176966 time to create 1 rle with new method : 1.3193094730377197 create new chi : 5.230076551437378 time to delete rle : 0.0014367103576660156 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 10309 TO DO : save crop sub photo not yet done ! save time : 0.9916501045227051 nb_obj : 13 nb_hashtags : 5 time to prepare the origin masks : 11.639927387237549 time for calcul the mask position with numpy : 0.8617184162139893 nb_pixel_total : 6325605 time to create 1 rle with new method : 1.0915946960449219 time for calcul the mask position with numpy : 0.03844165802001953 nb_pixel_total : 55894 time to create 1 rle with old method : 0.06933975219726562 time for calcul the mask position with numpy : 0.03781270980834961 nb_pixel_total : 36156 time to create 1 rle with old method : 0.03983926773071289 time for calcul the mask position with numpy : 0.03531026840209961 nb_pixel_total : 22684 time to create 1 rle with old method : 0.02611708641052246 time for calcul the mask position with numpy : 0.03870582580566406 nb_pixel_total : 33141 time to create 1 rle with old method : 0.03857731819152832 time for calcul the mask position with numpy : 0.02429676055908203 nb_pixel_total : 3119 time to create 1 rle with old method : 0.003504514694213867 time for calcul the mask position with numpy : 0.03794717788696289 nb_pixel_total : 10381 time to create 1 rle with old method : 0.012145519256591797 time for calcul the mask position with numpy : 0.026961326599121094 nb_pixel_total : 81632 time to create 1 rle with old method : 0.09572792053222656 time for calcul the mask position with numpy : 0.022803306579589844 nb_pixel_total : 7022 time to create 1 rle with old method : 0.007887125015258789 time for calcul the mask position with numpy : 0.02404165267944336 nb_pixel_total : 28386 time to create 1 rle with old method : 0.032064199447631836 time for calcul the mask position with numpy : 0.022416353225708008 nb_pixel_total : 82822 time to create 1 rle with old method : 0.09629058837890625 time for calcul the mask position with numpy : 0.023759841918945312 nb_pixel_total : 80921 time to create 1 rle with old method : 0.09418249130249023 time for calcul the mask position with numpy : 0.023163557052612305 nb_pixel_total : 39036 time to create 1 rle with old method : 0.04804849624633789 time for calcul the mask position with numpy : 0.038152217864990234 nb_pixel_total : 243441 time to create 1 rle with new method : 1.7646713256835938 create new chi : 4.7502734661102295 time to delete rle : 0.0010793209075927734 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 10140 TO DO : save crop sub photo not yet done ! save time : 0.6952745914459229 nb_obj : 15 nb_hashtags : 4 time to prepare the origin masks : 11.601382732391357 time for calcul the mask position with numpy : 0.8250067234039307 nb_pixel_total : 6359794 time to create 1 rle with new method : 1.081963062286377 time for calcul the mask position with numpy : 0.0327763557434082 nb_pixel_total : 57400 time to create 1 rle with old method : 0.06432628631591797 time for calcul the mask position with numpy : 0.03518342971801758 nb_pixel_total : 34103 time to create 1 rle with old method : 0.0507810115814209 time for calcul the mask position with numpy : 0.03957176208496094 nb_pixel_total : 1328 time to create 1 rle with old method : 0.002279996871948242 time for calcul the mask position with numpy : 0.037119388580322266 nb_pixel_total : 1441 time to create 1 rle with old method : 0.001661539077758789 time for calcul the mask position with numpy : 0.03453707695007324 nb_pixel_total : 2213 time to create 1 rle with old method : 0.002710103988647461 time for calcul the mask position with numpy : 0.03408169746398926 nb_pixel_total : 6667 time to create 1 rle with old method : 0.007604837417602539 time for calcul the mask position with numpy : 0.02160334587097168 nb_pixel_total : 8896 time to create 1 rle with old method : 0.010915994644165039 time for calcul the mask position with numpy : 0.023398399353027344 nb_pixel_total : 4310 time to create 1 rle with old method : 0.004977226257324219 time for calcul the mask position with numpy : 0.021591901779174805 nb_pixel_total : 5951 time to create 1 rle with old method : 0.006978750228881836 time for calcul the mask position with numpy : 0.022646188735961914 nb_pixel_total : 10598 time to create 1 rle with old method : 0.012668848037719727 time for calcul the mask position with numpy : 0.029462575912475586 nb_pixel_total : 409517 time to create 1 rle with new method : 0.8594448566436768 time for calcul the mask position with numpy : 0.02666187286376953 nb_pixel_total : 4162 time to create 1 rle with old method : 0.006742715835571289 time for calcul the mask position with numpy : 0.0378725528717041 nb_pixel_total : 19219 time to create 1 rle with old method : 0.028484821319580078 time for calcul the mask position with numpy : 0.036580801010131836 nb_pixel_total : 74927 time to create 1 rle with old method : 0.08387875556945801 time for calcul the mask position with numpy : 0.035607337951660156 nb_pixel_total : 49714 time to create 1 rle with old method : 0.05565690994262695 create new chi : 3.636967182159424 time to delete rle : 0.0009191036224365234 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 8543 TO DO : save crop sub photo not yet done ! save time : 0.7626392841339111 nb_obj : 10 nb_hashtags : 4 time to prepare the origin masks : 11.591517925262451 time for calcul the mask position with numpy : 1.0453252792358398 nb_pixel_total : 6539544 time to create 1 rle with new method : 1.1338233947753906 time for calcul the mask position with numpy : 0.022734642028808594 nb_pixel_total : 67635 time to create 1 rle with old method : 0.07973337173461914 time for calcul the mask position with numpy : 0.02139878273010254 nb_pixel_total : 126673 time to create 1 rle with old method : 0.14524030685424805 time for calcul the mask position with numpy : 0.021689653396606445 nb_pixel_total : 118331 time to create 1 rle with old method : 0.1371936798095703 time for calcul the mask position with numpy : 0.022481203079223633 nb_pixel_total : 22090 time to create 1 rle with old method : 0.024821043014526367 time for calcul the mask position with numpy : 0.0221710205078125 nb_pixel_total : 1502 time to create 1 rle with old method : 0.0021567344665527344 time for calcul the mask position with numpy : 0.021924734115600586 nb_pixel_total : 3882 time to create 1 rle with old method : 0.004383087158203125 time for calcul the mask position with numpy : 0.023663997650146484 nb_pixel_total : 3602 time to create 1 rle with old method : 0.0062448978424072266 time for calcul the mask position with numpy : 0.025332212448120117 nb_pixel_total : 3518 time to create 1 rle with old method : 0.004057645797729492 time for calcul the mask position with numpy : 0.02269458770751953 nb_pixel_total : 154761 time to create 1 rle with new method : 0.8007972240447998 time for calcul the mask position with numpy : 0.025548458099365234 nb_pixel_total : 8702 time to create 1 rle with old method : 0.010187387466430664 create new chi : 3.698082447052002 time to delete rle : 0.0011377334594726562 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 6723 TO DO : save crop sub photo not yet done ! save time : 0.9849224090576172 map_output_result : {1341265967: (0.0, 'Should be the crop_list due to order', 0), 1341265754: (0.0, 'Should be the crop_list due to order', 0), 1341265698: (0.0, 'Should be the crop_list due to order', 0), 1341265629: (0.0, 'Should be the crop_list due to order', 0), 1341265574: (0.0, 'Should be the crop_list due to order', 0), 1341265562: (0.0, 'Should be the crop_list due to order', 0), 1341265555: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1341265967, 1341265754, 1341265698, 1341265629, 1341265574, 1341265562, 1341265555] Looping around the photos to save general results len do output : 7 /1341265967.Didn't retrieve data . /1341265754.Didn't retrieve data . /1341265698.Didn't retrieve data . /1341265629.Didn't retrieve data . /1341265574.Didn't retrieve data . /1341265562.Didn't retrieve data . /1341265555.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265967', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265754', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265698', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265629', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265574', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265562', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265555', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 21 time used for this insertion : 0.013412714004516602 save_final save missing photos in datou_result : time spend for datou_step_exec : 124.60342955589294 time spend to save output : 0.015558719635009766 total time spend for step 5 : 124.61898827552795 step6:crop_condition Fri Mar 7 00:58:14 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 7 ! batch 1 Loaded 101 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741305496_287486 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8221206665039062 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741305498_287486 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9903936386108398 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741305502_287486 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.8062076568603516 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741305505_287486 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9238059520721436 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741305509_287486 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.388176679611206 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741305514_287486 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7002584934234619 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1341265967, 1341265754, 1341265698, 1341265629, 1341265574, 1341265562, 1341265555] Looping around the photos to save general results len do output : 18 /1341345240Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345243Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345244Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345247Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345248Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345249Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345250Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345251Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345254Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345255Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345256Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345262Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345263Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345264Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345265Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345266Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345267Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341345268Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265967', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265754', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265698', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265629', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265574', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265562', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265555', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 61 time used for this insertion : 0.023482084274291992 save_final save missing photos in datou_result : time spend for datou_step_exec : 22.258580207824707 time spend to save output : 0.024617433547973633 total time spend for step 6 : 22.28319764137268 step7:ventilate_hashtags_in_portfolio Fri Mar 7 00:58:36 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 21131115 get user id for portfolio 21131115 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131115 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('plastique','Carton_gris','environnement','kraft','Teint_Dans_La_Masse','papier','cartonnette','metal','Carton_brun','autre_refus','mal_croppe','flou')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131115 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('plastique','Carton_gris','environnement','kraft','Teint_Dans_La_Masse','papier','cartonnette','metal','Carton_brun','autre_refus','mal_croppe','flou')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/21131189,21131190,21131191,21131192,21131193,21131194,21131195,21131196,21131197,21131198,21131199,21131200?tags=mal_croppe,cartonnette,kraft,Teint_Dans_La_Masse,Carton_brun,metal,plastique,environnement,Carton_gris,autre_refus,papier,flou Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1341265967, 1341265754, 1341265698, 1341265629, 1341265574, 1341265562, 1341265555] Looping around the photos to save general results len do output : 1 /21131115. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265967', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265754', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265698', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265629', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265574', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265562', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265555', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 8 time used for this insertion : 0.013562917709350586 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.5085444450378418 time spend to save output : 0.013872623443603516 total time spend for step 7 : 1.5224170684814453 step8:final Fri Mar 7 00:58:38 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1341265967: ('0.06088626405858356',), 1341265754: ('0.06088626405858356',), 1341265698: ('0.06088626405858356',), 1341265629: ('0.06088626405858356',), 1341265574: ('0.06088626405858356',), 1341265562: ('0.06088626405858356',), 1341265555: ('0.06088626405858356',)} new output for save of step final : {1341265967: ('0.06088626405858356',), 1341265754: ('0.06088626405858356',), 1341265698: ('0.06088626405858356',), 1341265629: ('0.06088626405858356',), 1341265574: ('0.06088626405858356',), 1341265562: ('0.06088626405858356',), 1341265555: ('0.06088626405858356',)} [1341265967, 1341265754, 1341265698, 1341265629, 1341265574, 1341265562, 1341265555] Looping around the photos to save general results len do output : 7 /1341265967.Didn't retrieve data . /1341265754.Didn't retrieve data . /1341265698.Didn't retrieve data . /1341265629.Didn't retrieve data . /1341265574.Didn't retrieve data . /1341265562.Didn't retrieve data . /1341265555.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265967', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265754', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265698', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265629', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265574', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265562', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265555', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 21 time used for this insertion : 0.012691020965576172 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.1069636344909668 time spend to save output : 0.013013362884521484 total time spend for step 8 : 0.11997699737548828 step9:velours_tree Fri Mar 7 00:58:38 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.13591361045837402 time spend to save output : 4.6253204345703125e-05 total time spend for step 9 : 0.13595986366271973 step10:send_mail_cod Fri Mar 7 00:58:38 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21131115_07-03-2025_00_58_38.pdf 21131189 imagette211311891741305518 21131190 change filename to text .change filename to text .change filename to text .change filename to text .imagette211311901741305518 21131191 change filename to text .imagette211311911741305518 21131194 imagette211311941741305518 21131195 change filename to text .change filename to text .change filename to text .change filename to text .imagette211311951741305518 21131198 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211311981741305519 21131200 imagette211312001741305519 21131192 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211311921741305519 21131193 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211311931741305519 21131197 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211311971741305520 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21131115 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21131189,21131190,21131191,21131192,21131193,21131194,21131195,21131196,21131197,21131198,21131199,21131200?tags=mal_croppe,cartonnette,kraft,Teint_Dans_La_Masse,Carton_brun,metal,plastique,environnement,Carton_gris,autre_refus,papier,flou your option no_mail is active, we will not send the real mail to your client args[1341265967] : ('0.06088626405858356',) no score found for photo 1341265967 We are sending mail with results at report@fotonower.com args[1341265754] : ('0.06088626405858356',) no score found for photo 1341265754 We are sending mail with results at report@fotonower.com args[1341265698] : ('0.06088626405858356',) no score found for photo 1341265698 We are sending mail with results at report@fotonower.com args[1341265629] : ('0.06088626405858356',) no score found for photo 1341265629 We are sending mail with results at report@fotonower.com args[1341265574] : ('0.06088626405858356',) no score found for photo 1341265574 We are sending mail with results at report@fotonower.com args[1341265562] : ('0.06088626405858356',) no score found for photo 1341265562 We are sending mail with results at report@fotonower.com args[1341265555] : ('0.06088626405858356',) no score found for photo 1341265555 We are sending mail with results at report@fotonower.com refus_total : 0.06088626405858356 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=21131115 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1341265562,1341265574,1341265629,1341265698,1341265754,1341266213,1341265971,1341265555,1341265981,1341265987,1341265999,1341266186,1341265975,1341266194,1341266207,1341265967,1341266663) Found this number of photos: 17 begin to download photo : 1341265562 begin to download photo : 1341265754 begin to download photo : 1341265981 begin to download photo : 1341265975 begin to download photo : 1341266663 download finish for photo 1341265981 begin to download photo : 1341265987 download finish for photo 1341265562 begin to download photo : 1341265574 download finish for photo 1341265754 begin to download photo : 1341266213 download finish for photo 1341266663 download finish for photo 1341265975 begin to download photo : 1341266194 download finish for photo 1341265987 begin to download photo : 1341265999 download finish for photo 1341266213 begin to download photo : 1341265971 download finish for photo 1341265574 begin to download photo : 1341265629 download finish for photo 1341266194 begin to download photo : 1341266207 download finish for photo 1341265971 begin to download photo : 1341265555 download finish for photo 1341265629 begin to download photo : 1341265698 download finish for photo 1341265999 begin to download photo : 1341266186 download finish for photo 1341265698 download finish for photo 1341265555 download finish for photo 1341266207 begin to download photo : 1341265967 download finish for photo 1341266186 download finish for photo 1341265967 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131115_07-03-2025_00_58_38.pdf results_Auto_P21131115_07-03-2025_00_58_38.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131115_07-03-2025_00_58_38.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','21131115','results_Auto_P21131115_07-03-2025_00_58_38.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131115_07-03-2025_00_58_38.pdf','pdf','','0.7','0.06088626405858356') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1341265967, 1341265754, 1341265698, 1341265629, 1341265574, 1341265562, 1341265555] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265967', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265754', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265698', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265629', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265574', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265562', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265555', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 7 time used for this insertion : 0.011934518814086914 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.144833564758301 time spend to save output : 0.012287616729736328 total time spend for step 10 : 6.157121181488037 step11:split_time_score Fri Mar 7 00:58:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('17', 17),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 06032025 21131115 Nombre de photos uploadées : 17 / 23040 (0%) 06032025 21131115 Nombre de photos taguées (types de déchets): 0 / 17 (0%) 06032025 21131115 Nombre de photos taguées (volume) : 0 / 17 (0%) elapsed_time : load_data_split_time_score 1.9073486328125e-06 elapsed_time : order_list_meta_photo_and_scores 5.7220458984375e-06 ????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.000978708267211914 elapsed_time : insert_dashboard_record_day_entry 0.021744251251220703 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131086 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131087 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131088 order by id desc limit 1 Qualite : 0.08488825436881216 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21110324_06-03-2025_13_24_09.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21110324 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21110324 AND mptpi.`type`=3327 To do Qualite : 0.08456570303462264 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21110325_06-03-2025_09_27_50.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21110325 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21110325 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21110326 order by id desc limit 1 Qualite : 0.05850682993931322 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21110327_06-03-2025_09_32_24.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21110327 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21110327 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131089 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131090 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131091 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131092 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131093 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131094 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131095 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131096 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131097 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131098 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131099 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131100 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131101 order by id desc limit 1 Qualite : 0.09011573601023397 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131102_07-03-2025_00_33_16.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131102 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131102 AND mptpi.`type`=3327 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131103 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131104 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131105 order by id desc limit 1 Qualite : 0.1208628538400596 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131106_07-03-2025_00_22_01.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131106 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131106 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131107 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131108 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131109 order by id desc limit 1 Qualite : 0.23950699235455492 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131110_07-03-2025_00_45_00.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131110 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131110 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131111 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131112 order by id desc limit 1 Qualite : 0.09219659423566663 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131113_07-03-2025_00_21_45.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131113 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131113 AND mptpi.`type`=4207 To do Qualite : 0.22215740173384166 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131114_07-03-2025_00_29_29.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131114 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131114 AND mptpi.`type`=3594 To do Qualite : 0.0628226615711991 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131115_07-03-2025_00_58_38.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131115 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131115 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'06032025': {'nb_upload': 17, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1341265967, 1341265754, 1341265698, 1341265629, 1341265574, 1341265562, 1341265555] Looping around the photos to save general results len do output : 1 /21131115Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265967', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265754', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265698', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265629', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265574', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265562', None, None, None, None, None, '2646484') ('3459', None, None, None, None, None, None, None, '2646484') ('3459', None, '1341265555', None, None, None, None, None, '2646484') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 8 time used for this insertion : 0.013536214828491211 save_final save missing photos in datou_result : time spend for datou_step_exec : 20.800818920135498 time spend to save output : 0.013782739639282227 total time spend for step 11 : 20.81460165977478 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 2 len(input) + len(total_photo_id_missing) : 7 set_done_treatment 103.80user 132.52system 4:42.19elapsed 83%CPU (0avgtext+0avgdata 3589036maxresident)k 2877344inputs+90120outputs (18086major+9457679minor)pagefaults 0swaps