python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 121508 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 21>10 please execute split_portfolio.py -i 21131106 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2646475'] with mtr_portfolio_ids : ['21131106'] and first list_photo_ids : [] new path : /proc/121508/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.5711939334869385 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Mar 7 00:14:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4386 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-07 00:14:32.767795: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-07 00:14:32.795204: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-07 00:14:32.797049: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8b60000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-07 00:14:32.797079: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-07 00:14:32.801084: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-07 00:14:33.067377: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x24a3d860 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-07 00:14:33.067443: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-07 00:14:33.068690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 00:14:33.069171: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 00:14:33.074991: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 00:14:33.078098: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 00:14:33.079418: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 00:14:33.081961: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 00:14:33.083238: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 00:14:33.088422: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 00:14:33.089589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 00:14:33.089676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 00:14:33.090251: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-07 00:14:33.090267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-07 00:14:33.090275: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-07 00:14:33.091249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3934 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-07 00:14:33.412439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 00:14:33.412570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 00:14:33.412598: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 00:14:33.412623: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 00:14:33.412647: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 00:14:33.412671: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 00:14:33.412694: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 00:14:33.412718: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 00:14:33.413964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 00:14:33.415396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 00:14:33.415443: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 00:14:33.415468: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 00:14:33.415491: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 00:14:33.415514: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 00:14:33.415537: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 00:14:33.415567: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 00:14:33.415592: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 00:14:33.416824: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 00:14:33.416864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-07 00:14:33.416877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-07 00:14:33.416888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-07 00:14:33.418120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3934 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-07 00:14:46.551148: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 00:14:46.722151: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 00:14:48.371146: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.183374: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-03-07 00:14:49.254638: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.254718: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:14:49.255777: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.255801: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:14:49.262876: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.262943: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:14:49.263928: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.263947: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:14:49.294734: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.294806: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 19.91MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:14:49.294821: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-03-07 00:14:49.295886: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.295927: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:14:49.296958: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.296977: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:14:49.304350: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.304397: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:14:49.305417: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.305436: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:14:49.306470: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.306488: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-07 00:14:49.307521: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.316300: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.317365: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.336441: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.337130: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.337783: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.338414: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.342530: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.343186: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.343824: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.344472: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.345576: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.355800: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.356438: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.365552: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.366195: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.366809: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.367451: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.368097: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-07 00:14:49.368705: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.34G (3590193152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 23 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.74009 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 30 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 146.45001 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 21 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 27 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 21 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 10 Detection mask done ! Trying to reset tf kernel 121744 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3193 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4386 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0030336380004882812 nb_pixel_total : 134094 time to create 1 rle with old method : 0.16498923301696777 length of segment : 458 time for calcul the mask position with numpy : 0.0006272792816162109 nb_pixel_total : 26537 time to create 1 rle with old method : 0.03851580619812012 length of segment : 169 time for calcul the mask position with numpy : 0.0025594234466552734 nb_pixel_total : 139243 time to create 1 rle with old method : 0.19637799263000488 length of segment : 340 time for calcul the mask position with numpy : 0.015219926834106445 nb_pixel_total : 593590 time to create 1 rle with new method : 0.6237468719482422 length of segment : 1037 time for calcul the mask position with numpy : 0.00022172927856445312 nb_pixel_total : 7997 time to create 1 rle with old method : 0.009451627731323242 length of segment : 79 time for calcul the mask position with numpy : 0.0015854835510253906 nb_pixel_total : 98838 time to create 1 rle with old method : 0.11225295066833496 length of segment : 487 time for calcul the mask position with numpy : 0.00045180320739746094 nb_pixel_total : 12255 time to create 1 rle with old method : 0.015054464340209961 length of segment : 229 time for calcul the mask position with numpy : 0.003714323043823242 nb_pixel_total : 202933 time to create 1 rle with new method : 0.007845163345336914 length of segment : 628 time for calcul the mask position with numpy : 0.0008840560913085938 nb_pixel_total : 44798 time to create 1 rle with old method : 0.056818485260009766 length of segment : 255 time for calcul the mask position with numpy : 0.001016378402709961 nb_pixel_total : 55159 time to create 1 rle with old method : 0.062896728515625 length of segment : 490 time for calcul the mask position with numpy : 0.0008273124694824219 nb_pixel_total : 36065 time to create 1 rle with old method : 0.04480433464050293 length of segment : 222 time for calcul the mask position with numpy : 0.0022764205932617188 nb_pixel_total : 117991 time to create 1 rle with old method : 0.1408522129058838 length of segment : 513 time for calcul the mask position with numpy : 0.0024309158325195312 nb_pixel_total : 40863 time to create 1 rle with old method : 0.07645869255065918 length of segment : 360 time for calcul the mask position with numpy : 0.0002460479736328125 nb_pixel_total : 3147 time to create 1 rle with old method : 0.003949880599975586 length of segment : 65 time for calcul the mask position with numpy : 0.0005223751068115234 nb_pixel_total : 13586 time to create 1 rle with old method : 0.01724982261657715 length of segment : 118 time for calcul the mask position with numpy : 0.0011720657348632812 nb_pixel_total : 52332 time to create 1 rle with old method : 0.05856680870056152 length of segment : 375 time for calcul the mask position with numpy : 0.004902839660644531 nb_pixel_total : 211310 time to create 1 rle with new method : 0.007173061370849609 length of segment : 528 time for calcul the mask position with numpy : 0.0020301342010498047 nb_pixel_total : 51857 time to create 1 rle with old method : 0.059880733489990234 length of segment : 319 time for calcul the mask position with numpy : 0.001758575439453125 nb_pixel_total : 26457 time to create 1 rle with old method : 0.033493757247924805 length of segment : 258 time for calcul the mask position with numpy : 0.5907397270202637 nb_pixel_total : 5038824 time to create 1 rle with new method : 1.0500907897949219 length of segment : 4044 time for calcul the mask position with numpy : 0.003021717071533203 nb_pixel_total : 198564 time to create 1 rle with new method : 0.004988670349121094 length of segment : 494 time for calcul the mask position with numpy : 0.0021789073944091797 nb_pixel_total : 59308 time to create 1 rle with old method : 0.06821227073669434 length of segment : 551 time for calcul the mask position with numpy : 0.00032520294189453125 nb_pixel_total : 6057 time to create 1 rle with old method : 0.007503509521484375 length of segment : 97 time for calcul the mask position with numpy : 0.0005128383636474609 nb_pixel_total : 11648 time to create 1 rle with old method : 0.013892173767089844 length of segment : 124 time for calcul the mask position with numpy : 0.0004818439483642578 nb_pixel_total : 17493 time to create 1 rle with old method : 0.0202178955078125 length of segment : 147 time for calcul the mask position with numpy : 0.015477895736694336 nb_pixel_total : 467210 time to create 1 rle with new method : 0.18295001983642578 length of segment : 978 time for calcul the mask position with numpy : 0.00014352798461914062 nb_pixel_total : 1081 time to create 1 rle with old method : 0.0013954639434814453 length of segment : 46 time for calcul the mask position with numpy : 0.001789093017578125 nb_pixel_total : 74886 time to create 1 rle with old method : 0.08989429473876953 length of segment : 311 time for calcul the mask position with numpy : 0.008112668991088867 nb_pixel_total : 304883 time to create 1 rle with new method : 0.012173652648925781 length of segment : 606 time for calcul the mask position with numpy : 0.00026726722717285156 nb_pixel_total : 2586 time to create 1 rle with old method : 0.0032949447631835938 length of segment : 54 time for calcul the mask position with numpy : 0.0031824111938476562 nb_pixel_total : 94714 time to create 1 rle with old method : 0.1078488826751709 length of segment : 503 time for calcul the mask position with numpy : 0.0011789798736572266 nb_pixel_total : 77646 time to create 1 rle with old method : 0.08813142776489258 length of segment : 377 time for calcul the mask position with numpy : 0.002220630645751953 nb_pixel_total : 97635 time to create 1 rle with old method : 0.11158895492553711 length of segment : 553 time for calcul the mask position with numpy : 0.002624988555908203 nb_pixel_total : 76064 time to create 1 rle with old method : 0.08974862098693848 length of segment : 412 time for calcul the mask position with numpy : 0.00684356689453125 nb_pixel_total : 248482 time to create 1 rle with new method : 0.010411977767944336 length of segment : 594 time for calcul the mask position with numpy : 0.001184701919555664 nb_pixel_total : 68834 time to create 1 rle with old method : 0.08910298347473145 length of segment : 332 time for calcul the mask position with numpy : 0.0016231536865234375 nb_pixel_total : 58274 time to create 1 rle with old method : 0.06692051887512207 length of segment : 329 time for calcul the mask position with numpy : 0.0007908344268798828 nb_pixel_total : 21149 time to create 1 rle with old method : 0.025009870529174805 length of segment : 313 time for calcul the mask position with numpy : 0.0015306472778320312 nb_pixel_total : 46912 time to create 1 rle with old method : 0.058837175369262695 length of segment : 173 time for calcul the mask position with numpy : 0.002388477325439453 nb_pixel_total : 57446 time to create 1 rle with old method : 0.07013773918151855 length of segment : 202 time for calcul the mask position with numpy : 0.007126569747924805 nb_pixel_total : 254083 time to create 1 rle with new method : 0.010941028594970703 length of segment : 610 time for calcul the mask position with numpy : 0.001790761947631836 nb_pixel_total : 52200 time to create 1 rle with old method : 0.05892324447631836 length of segment : 210 time for calcul the mask position with numpy : 0.001093149185180664 nb_pixel_total : 29364 time to create 1 rle with old method : 0.03456687927246094 length of segment : 366 time for calcul the mask position with numpy : 0.0034911632537841797 nb_pixel_total : 74916 time to create 1 rle with old method : 0.08530950546264648 length of segment : 546 time for calcul the mask position with numpy : 0.0034351348876953125 nb_pixel_total : 82527 time to create 1 rle with old method : 0.09856224060058594 length of segment : 483 time for calcul the mask position with numpy : 0.002605438232421875 nb_pixel_total : 117014 time to create 1 rle with old method : 0.15662527084350586 length of segment : 402 time for calcul the mask position with numpy : 0.0025131702423095703 nb_pixel_total : 85158 time to create 1 rle with old method : 0.09459304809570312 length of segment : 433 time for calcul the mask position with numpy : 0.0034685134887695312 nb_pixel_total : 117635 time to create 1 rle with old method : 0.13243842124938965 length of segment : 547 time for calcul the mask position with numpy : 0.0001595020294189453 nb_pixel_total : 1564 time to create 1 rle with old method : 0.002111673355102539 length of segment : 31 time for calcul the mask position with numpy : 0.0007231235504150391 nb_pixel_total : 27540 time to create 1 rle with old method : 0.031178951263427734 length of segment : 380 time for calcul the mask position with numpy : 0.006909370422363281 nb_pixel_total : 198276 time to create 1 rle with new method : 0.015916109085083008 length of segment : 938 time for calcul the mask position with numpy : 0.002365589141845703 nb_pixel_total : 81216 time to create 1 rle with old method : 0.09266853332519531 length of segment : 373 time for calcul the mask position with numpy : 0.0027310848236083984 nb_pixel_total : 112699 time to create 1 rle with old method : 0.1264207363128662 length of segment : 499 time for calcul the mask position with numpy : 0.0023703575134277344 nb_pixel_total : 29725 time to create 1 rle with old method : 0.03446650505065918 length of segment : 367 time for calcul the mask position with numpy : 0.0008308887481689453 nb_pixel_total : 15550 time to create 1 rle with old method : 0.01808309555053711 length of segment : 232 time for calcul the mask position with numpy : 0.002343893051147461 nb_pixel_total : 116766 time to create 1 rle with old method : 0.15639805793762207 length of segment : 378 time for calcul the mask position with numpy : 0.0014507770538330078 nb_pixel_total : 38502 time to create 1 rle with old method : 0.04518747329711914 length of segment : 232 time for calcul the mask position with numpy : 0.0012962818145751953 nb_pixel_total : 30140 time to create 1 rle with old method : 0.0364227294921875 length of segment : 279 time for calcul the mask position with numpy : 0.0017764568328857422 nb_pixel_total : 34557 time to create 1 rle with old method : 0.039647579193115234 length of segment : 340 time for calcul the mask position with numpy : 0.0005755424499511719 nb_pixel_total : 12100 time to create 1 rle with old method : 0.014035463333129883 length of segment : 115 time for calcul the mask position with numpy : 0.003622293472290039 nb_pixel_total : 116603 time to create 1 rle with old method : 0.1334362030029297 length of segment : 439 time for calcul the mask position with numpy : 0.003998756408691406 nb_pixel_total : 94699 time to create 1 rle with old method : 0.11082696914672852 length of segment : 359 time for calcul the mask position with numpy : 0.0012145042419433594 nb_pixel_total : 26955 time to create 1 rle with old method : 0.0313260555267334 length of segment : 280 time for calcul the mask position with numpy : 0.0035974979400634766 nb_pixel_total : 87078 time to create 1 rle with old method : 0.0999608039855957 length of segment : 535 time for calcul the mask position with numpy : 0.0006124973297119141 nb_pixel_total : 12757 time to create 1 rle with old method : 0.015938520431518555 length of segment : 125 time for calcul the mask position with numpy : 0.004526615142822266 nb_pixel_total : 74322 time to create 1 rle with old method : 0.08361196517944336 length of segment : 669 time for calcul the mask position with numpy : 0.0010716915130615234 nb_pixel_total : 35210 time to create 1 rle with old method : 0.04456067085266113 length of segment : 189 time for calcul the mask position with numpy : 0.003299713134765625 nb_pixel_total : 54574 time to create 1 rle with old method : 0.06273007392883301 length of segment : 470 time for calcul the mask position with numpy : 0.0022368431091308594 nb_pixel_total : 42432 time to create 1 rle with old method : 0.05106234550476074 length of segment : 395 time for calcul the mask position with numpy : 0.0001709461212158203 nb_pixel_total : 2058 time to create 1 rle with old method : 0.002688169479370117 length of segment : 45 time for calcul the mask position with numpy : 0.0037508010864257812 nb_pixel_total : 153273 time to create 1 rle with new method : 0.006245851516723633 length of segment : 466 time for calcul the mask position with numpy : 0.0016846656799316406 nb_pixel_total : 44812 time to create 1 rle with old method : 0.0497746467590332 length of segment : 275 time for calcul the mask position with numpy : 0.0004334449768066406 nb_pixel_total : 18105 time to create 1 rle with old method : 0.02298903465270996 length of segment : 126 time for calcul the mask position with numpy : 0.004784345626831055 nb_pixel_total : 204041 time to create 1 rle with new method : 0.008323192596435547 length of segment : 441 time for calcul the mask position with numpy : 0.0010235309600830078 nb_pixel_total : 41663 time to create 1 rle with old method : 0.04973483085632324 length of segment : 426 time for calcul the mask position with numpy : 0.0005390644073486328 nb_pixel_total : 14616 time to create 1 rle with old method : 0.01804184913635254 length of segment : 167 time for calcul the mask position with numpy : 0.0009095668792724609 nb_pixel_total : 42826 time to create 1 rle with old method : 0.0484004020690918 length of segment : 311 time for calcul the mask position with numpy : 0.0011744499206542969 nb_pixel_total : 29239 time to create 1 rle with old method : 0.03288435935974121 length of segment : 264 time for calcul the mask position with numpy : 0.001959562301635742 nb_pixel_total : 74710 time to create 1 rle with old method : 0.08272838592529297 length of segment : 649 time for calcul the mask position with numpy : 0.0015842914581298828 nb_pixel_total : 56926 time to create 1 rle with old method : 0.06335926055908203 length of segment : 466 time for calcul the mask position with numpy : 0.001195669174194336 nb_pixel_total : 41296 time to create 1 rle with old method : 0.04651045799255371 length of segment : 516 time for calcul the mask position with numpy : 0.0017261505126953125 nb_pixel_total : 51388 time to create 1 rle with old method : 0.0643911361694336 length of segment : 160 time for calcul the mask position with numpy : 0.0009591579437255859 nb_pixel_total : 19591 time to create 1 rle with old method : 0.022640228271484375 length of segment : 327 time for calcul the mask position with numpy : 0.004180431365966797 nb_pixel_total : 215560 time to create 1 rle with new method : 0.011400699615478516 length of segment : 452 time for calcul the mask position with numpy : 0.0022513866424560547 nb_pixel_total : 64832 time to create 1 rle with old method : 0.07313275337219238 length of segment : 213 time for calcul the mask position with numpy : 0.032025814056396484 nb_pixel_total : 996141 time to create 1 rle with new method : 1.2331383228302002 length of segment : 1290 time for calcul the mask position with numpy : 0.00022125244140625 nb_pixel_total : 4059 time to create 1 rle with old method : 0.004934787750244141 length of segment : 122 time for calcul the mask position with numpy : 0.0012221336364746094 nb_pixel_total : 25354 time to create 1 rle with old method : 0.03103494644165039 length of segment : 267 time for calcul the mask position with numpy : 0.0012388229370117188 nb_pixel_total : 44227 time to create 1 rle with old method : 0.05093812942504883 length of segment : 275 time for calcul the mask position with numpy : 0.005506277084350586 nb_pixel_total : 154597 time to create 1 rle with new method : 0.008372068405151367 length of segment : 563 time for calcul the mask position with numpy : 0.0004112720489501953 nb_pixel_total : 10942 time to create 1 rle with old method : 0.017908096313476562 length of segment : 147 time for calcul the mask position with numpy : 0.0003151893615722656 nb_pixel_total : 8775 time to create 1 rle with old method : 0.01469564437866211 length of segment : 79 time for calcul the mask position with numpy : 0.004331827163696289 nb_pixel_total : 135289 time to create 1 rle with old method : 0.15008234977722168 length of segment : 363 time for calcul the mask position with numpy : 0.00014138221740722656 nb_pixel_total : 1648 time to create 1 rle with old method : 0.002179384231567383 length of segment : 59 time for calcul the mask position with numpy : 0.003047943115234375 nb_pixel_total : 103302 time to create 1 rle with old method : 0.1195070743560791 length of segment : 506 time for calcul the mask position with numpy : 0.0013294219970703125 nb_pixel_total : 39714 time to create 1 rle with old method : 0.04534006118774414 length of segment : 319 time for calcul the mask position with numpy : 0.005098581314086914 nb_pixel_total : 167002 time to create 1 rle with new method : 0.01067805290222168 length of segment : 901 time for calcul the mask position with numpy : 0.003331899642944336 nb_pixel_total : 124817 time to create 1 rle with old method : 0.15671706199645996 length of segment : 685 time for calcul the mask position with numpy : 0.0003802776336669922 nb_pixel_total : 7280 time to create 1 rle with old method : 0.009671926498413086 length of segment : 109 time for calcul the mask position with numpy : 0.0004382133483886719 nb_pixel_total : 4551 time to create 1 rle with old method : 0.005717039108276367 length of segment : 141 time for calcul the mask position with numpy : 0.001720428466796875 nb_pixel_total : 57519 time to create 1 rle with old method : 0.07278680801391602 length of segment : 336 time for calcul the mask position with numpy : 0.0010151863098144531 nb_pixel_total : 27164 time to create 1 rle with old method : 0.03305220603942871 length of segment : 217 time for calcul the mask position with numpy : 0.008519887924194336 nb_pixel_total : 197994 time to create 1 rle with new method : 0.014137983322143555 length of segment : 636 time for calcul the mask position with numpy : 0.00260162353515625 nb_pixel_total : 57734 time to create 1 rle with old method : 0.06832599639892578 length of segment : 461 time for calcul the mask position with numpy : 0.0014824867248535156 nb_pixel_total : 33555 time to create 1 rle with old method : 0.04214310646057129 length of segment : 365 time for calcul the mask position with numpy : 0.0004706382751464844 nb_pixel_total : 7027 time to create 1 rle with old method : 0.00877070426940918 length of segment : 96 time for calcul the mask position with numpy : 0.0011563301086425781 nb_pixel_total : 22260 time to create 1 rle with old method : 0.028202056884765625 length of segment : 259 time for calcul the mask position with numpy : 0.0003979206085205078 nb_pixel_total : 4052 time to create 1 rle with old method : 0.004961490631103516 length of segment : 126 time for calcul the mask position with numpy : 0.00040221214294433594 nb_pixel_total : 11597 time to create 1 rle with old method : 0.01395416259765625 length of segment : 144 time for calcul the mask position with numpy : 0.0021920204162597656 nb_pixel_total : 71413 time to create 1 rle with old method : 0.08526158332824707 length of segment : 486 time for calcul the mask position with numpy : 0.00041747093200683594 nb_pixel_total : 6475 time to create 1 rle with old method : 0.008040428161621094 length of segment : 122 time for calcul the mask position with numpy : 0.008565187454223633 nb_pixel_total : 309263 time to create 1 rle with new method : 0.014290332794189453 length of segment : 1099 time for calcul the mask position with numpy : 0.0024378299713134766 nb_pixel_total : 85918 time to create 1 rle with old method : 0.10074615478515625 length of segment : 421 time for calcul the mask position with numpy : 0.00046753883361816406 nb_pixel_total : 9463 time to create 1 rle with old method : 0.011666059494018555 length of segment : 152 time for calcul the mask position with numpy : 0.014532089233398438 nb_pixel_total : 428834 time to create 1 rle with new method : 0.027547359466552734 length of segment : 1303 time for calcul the mask position with numpy : 0.002471923828125 nb_pixel_total : 46067 time to create 1 rle with old method : 0.05828404426574707 length of segment : 265 time for calcul the mask position with numpy : 0.0003085136413574219 nb_pixel_total : 6689 time to create 1 rle with old method : 0.007891416549682617 length of segment : 125 time for calcul the mask position with numpy : 0.0012054443359375 nb_pixel_total : 33933 time to create 1 rle with old method : 0.055588483810424805 length of segment : 209 time for calcul the mask position with numpy : 0.0005340576171875 nb_pixel_total : 27779 time to create 1 rle with old method : 0.03275418281555176 length of segment : 157 time for calcul the mask position with numpy : 0.0009741783142089844 nb_pixel_total : 43380 time to create 1 rle with old method : 0.05257296562194824 length of segment : 294 time for calcul the mask position with numpy : 0.002354145050048828 nb_pixel_total : 42049 time to create 1 rle with old method : 0.04982924461364746 length of segment : 344 time for calcul the mask position with numpy : 0.004383087158203125 nb_pixel_total : 153311 time to create 1 rle with new method : 0.0066373348236083984 length of segment : 493 time for calcul the mask position with numpy : 0.0003483295440673828 nb_pixel_total : 5633 time to create 1 rle with old method : 0.007195234298706055 length of segment : 81 time for calcul the mask position with numpy : 0.003982067108154297 nb_pixel_total : 167385 time to create 1 rle with new method : 0.006512165069580078 length of segment : 616 time for calcul the mask position with numpy : 0.0007648468017578125 nb_pixel_total : 29396 time to create 1 rle with old method : 0.03468513488769531 length of segment : 172 time for calcul the mask position with numpy : 0.0005033016204833984 nb_pixel_total : 9425 time to create 1 rle with old method : 0.011994600296020508 length of segment : 113 time for calcul the mask position with numpy : 0.0017311573028564453 nb_pixel_total : 85528 time to create 1 rle with old method : 0.09843158721923828 length of segment : 412 time for calcul the mask position with numpy : 0.0013141632080078125 nb_pixel_total : 22307 time to create 1 rle with old method : 0.027568817138671875 length of segment : 363 time for calcul the mask position with numpy : 0.0040280818939208984 nb_pixel_total : 124008 time to create 1 rle with old method : 0.1477823257446289 length of segment : 342 time for calcul the mask position with numpy : 0.0022149085998535156 nb_pixel_total : 39436 time to create 1 rle with old method : 0.04776883125305176 length of segment : 252 time for calcul the mask position with numpy : 0.0028290748596191406 nb_pixel_total : 61453 time to create 1 rle with old method : 0.06938791275024414 length of segment : 336 time for calcul the mask position with numpy : 0.010345220565795898 nb_pixel_total : 200424 time to create 1 rle with new method : 0.023436784744262695 length of segment : 682 time for calcul the mask position with numpy : 0.0003714561462402344 nb_pixel_total : 1778 time to create 1 rle with old method : 0.0023958683013916016 length of segment : 79 time for calcul the mask position with numpy : 0.0007297992706298828 nb_pixel_total : 4422 time to create 1 rle with old method : 0.007901191711425781 length of segment : 125 time for calcul the mask position with numpy : 0.0012524127960205078 nb_pixel_total : 15720 time to create 1 rle with old method : 0.029458045959472656 length of segment : 105 time for calcul the mask position with numpy : 0.01109004020690918 nb_pixel_total : 186381 time to create 1 rle with new method : 0.017887592315673828 length of segment : 743 time for calcul the mask position with numpy : 0.0009942054748535156 nb_pixel_total : 35748 time to create 1 rle with old method : 0.04341745376586914 length of segment : 337 time for calcul the mask position with numpy : 0.0010457038879394531 nb_pixel_total : 20105 time to create 1 rle with old method : 0.023410320281982422 length of segment : 196 time for calcul the mask position with numpy : 0.0009098052978515625 nb_pixel_total : 20566 time to create 1 rle with old method : 0.024561166763305664 length of segment : 191 time for calcul the mask position with numpy : 0.005954265594482422 nb_pixel_total : 130660 time to create 1 rle with old method : 0.1530144214630127 length of segment : 623 time for calcul the mask position with numpy : 0.0026750564575195312 nb_pixel_total : 104610 time to create 1 rle with old method : 0.1405181884765625 length of segment : 398 time for calcul the mask position with numpy : 0.0030510425567626953 nb_pixel_total : 146059 time to create 1 rle with old method : 0.1700296401977539 length of segment : 579 time for calcul the mask position with numpy : 0.0006263256072998047 nb_pixel_total : 25811 time to create 1 rle with old method : 0.02873516082763672 length of segment : 201 time for calcul the mask position with numpy : 0.0004639625549316406 nb_pixel_total : 15821 time to create 1 rle with old method : 0.018879413604736328 length of segment : 224 time for calcul the mask position with numpy : 0.0010118484497070312 nb_pixel_total : 31917 time to create 1 rle with old method : 0.03723788261413574 length of segment : 206 time for calcul the mask position with numpy : 0.00029158592224121094 nb_pixel_total : 16010 time to create 1 rle with old method : 0.01896214485168457 length of segment : 216 time for calcul the mask position with numpy : 0.002238035202026367 nb_pixel_total : 81592 time to create 1 rle with old method : 0.08923220634460449 length of segment : 361 time for calcul the mask position with numpy : 0.6890475749969482 nb_pixel_total : 3907468 time to create 1 rle with new method : 1.555408000946045 length of segment : 3693 time for calcul the mask position with numpy : 0.019167184829711914 nb_pixel_total : 1331929 time to create 1 rle with new method : 0.3343536853790283 length of segment : 1652 time for calcul the mask position with numpy : 0.0008485317230224609 nb_pixel_total : 51531 time to create 1 rle with old method : 0.05960416793823242 length of segment : 425 time for calcul the mask position with numpy : 0.00038123130798339844 nb_pixel_total : 25686 time to create 1 rle with old method : 0.029425382614135742 length of segment : 139 time for calcul the mask position with numpy : 0.0005421638488769531 nb_pixel_total : 17968 time to create 1 rle with old method : 0.024956464767456055 length of segment : 228 time for calcul the mask position with numpy : 0.0013058185577392578 nb_pixel_total : 23528 time to create 1 rle with old method : 0.027263641357421875 length of segment : 239 time for calcul the mask position with numpy : 0.0014781951904296875 nb_pixel_total : 49328 time to create 1 rle with old method : 0.06970763206481934 length of segment : 364 time for calcul the mask position with numpy : 0.0010797977447509766 nb_pixel_total : 43191 time to create 1 rle with old method : 0.05094456672668457 length of segment : 259 time for calcul the mask position with numpy : 0.0012950897216796875 nb_pixel_total : 50170 time to create 1 rle with old method : 0.057415008544921875 length of segment : 269 time for calcul the mask position with numpy : 0.0016427040100097656 nb_pixel_total : 79900 time to create 1 rle with old method : 0.09392857551574707 length of segment : 412 time for calcul the mask position with numpy : 0.0018520355224609375 nb_pixel_total : 84015 time to create 1 rle with old method : 0.14367079734802246 length of segment : 412 time for calcul the mask position with numpy : 0.00039768218994140625 nb_pixel_total : 6757 time to create 1 rle with old method : 0.007939815521240234 length of segment : 150 time for calcul the mask position with numpy : 0.0029621124267578125 nb_pixel_total : 198053 time to create 1 rle with new method : 0.007273435592651367 length of segment : 471 time for calcul the mask position with numpy : 0.002166748046875 nb_pixel_total : 69176 time to create 1 rle with old method : 0.0759742259979248 length of segment : 519 time spent for convertir_results : 27.958121299743652 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 161 chid ids of type : 3663 Number RLEs to save : 65817 save missing photos in datou_result : time spend for datou_step_exec : 97.57799124717712 time spend to save output : 4.675484657287598 total time spend for step 1 : 102.25347590446472 step2:crop_condition Fri Mar 7 00:16:11 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 161 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 17 About to insert : list_path_to_insert length 17 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 65 About to insert : list_path_to_insert length 65 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1341243040, 1341242973, 1341242970, 1341242967, 1341242380, 1341242375, 1341241949, 1341241752, 1341241599, 1341241565] Looping around the photos to save general results len do output : 112 /-3696599587Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599606Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599629Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599621Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599623Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599624Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599622Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599673Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599678Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599667Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599687Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599700Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599690Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599685Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599702Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599701Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599721Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599598Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599610Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599614Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599611Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599636Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599649Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599664Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599671Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599712Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599735Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599590Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599615Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599630Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599638Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599646Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599666Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599662Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599703Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599698Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599697Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599723Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599599Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599585Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599593Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599588Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599591Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599600Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599592Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599603Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599619Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599613Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599604Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599617Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599628Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599626Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599633Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599635Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599637Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599645Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599648Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599663Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599660Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599665Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599644Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599641Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599658Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599643Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599657Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599640Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599656Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599653Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599651Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599650Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599655Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599642Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599680Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599684Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599679Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599683Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599677Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599675Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599704Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599692Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599709Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599706Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599686Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599691Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599689Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599710Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599722Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599728Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599724Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599718Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599725Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599717Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599729Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599713Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599727Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599732Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599733Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599742Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599739Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599736Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599743Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599737Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599740Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599605Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599625Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599654Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599674Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599693Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599696Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599594Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599618Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696599659Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341243040', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242973', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242970', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242967', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242380', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242375', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241949', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241752', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241599', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241565', None, None, None, None, None, '2646475') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 346 time used for this insertion : 0.03804183006286621 save_final save missing photos in datou_result : time spend for datou_step_exec : 52.9754204750061 time spend to save output : 0.041396141052246094 total time spend for step 2 : 53.01681661605835 step3:thcl Fri Mar 7 00:17:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.008756875991821289 time to convert the images to numpy array : 0.3109753131866455 time to import caffe and check if the image exist : 0.019345760345458984 time to convert the images to numpy array : 0.6536753177642822 time to import caffe and check if the image exist : 0.007477283477783203 time to convert the images to numpy array : 0.7683515548706055 time to import caffe and check if the image exist : 0.021548986434936523 time to convert the images to numpy array : 0.768582820892334 time to import caffe and check if the image exist : 0.034740447998046875 time to convert the images to numpy array : 0.7686977386474609 time to import caffe and check if the image exist : 0.019193649291992188 time to convert the images to numpy array : 1.06634521484375 time to import caffe and check if the image exist : 0.018359661102294922 time to convert the images to numpy array : 1.0686705112457275 time to import caffe and check if the image exist : 0.26532673835754395 time to convert the images to numpy array : 0.8229320049285889 time to import caffe and check if the image exist : 0.018803834915161133 time to convert the images to numpy array : 1.0763545036315918 time to import caffe and check if the image exist : 0.02611517906188965 time to convert the images to numpy array : 1.0965805053710938 total time to convert the images to numpy array : 1.3198254108428955 list photo_ids error: [] list photo_ids correct : [-3696599696, -3696599594, -3696599618, -3696599659, -3696599733, -3696599742, -3696599739, -3696599736, -3696599743, -3696599737, -3696599740, -3696599605, -3696599625, -3696599654, -3696599674, -3696599693, -3696599587, -3696599606, -3696599629, -3696599621, -3696599623, -3696599624, -3696599622, -3696599673, -3696599678, -3696599667, -3696599687, -3696599700, -3696599644, -3696599641, -3696599658, -3696599643, -3696599657, -3696599640, -3696599656, -3696599653, -3696599651, -3696599650, -3696599655, -3696599642, -3696599689, -3696599710, -3696599722, -3696599728, -3696599724, -3696599718, -3696599725, -3696599717, -3696599729, -3696599713, -3696599727, -3696599732, -3696599604, -3696599617, -3696599628, -3696599626, -3696599633, -3696599635, -3696599637, -3696599645, -3696599648, -3696599663, -3696599660, -3696599665, -3696599680, -3696599684, -3696599679, -3696599683, -3696599677, -3696599675, -3696599704, -3696599692, -3696599709, -3696599706, -3696599686, -3696599691, -3696599690, -3696599685, -3696599702, -3696599701, -3696599721, -3696599598, -3696599610, -3696599614, -3696599611, -3696599636, -3696599649, -3696599664, -3696599671, -3696599712, -3696599735, -3696599590, -3696599615, -3696599630, -3696599638, -3696599646, -3696599666, -3696599662, -3696599703, -3696599698, -3696599697, -3696599723, -3696599599, -3696599585, -3696599593, -3696599588, -3696599591, -3696599600, -3696599592, -3696599603, -3696599619, -3696599613] number of photos to traite : 112 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3193 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2974 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.7181222438812256 time used to do the prediction : 0.24138951301574707 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2972 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2972 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.6608524322509766 time used to do the prediction : 0.22319579124450684 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.7881393432617188e-05 save missing photos in datou_result : time spend for datou_step_exec : 12.418709516525269 time spend to save output : 0.0007722377777099609 total time spend for step 3 : 12.419481754302979 step4:merge_mask_thcl_custom Fri Mar 7 00:17:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 161 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1341243040, 1341242973, 1341242970, 1341242967, 1341242380, 1341242375, 1341241949, 1341241752, 1341241599, 1341241565] Looping around the photos to save general results len do output : 10 /1341243040Didn't retrieve data .Didn't retrieve data . /1341242973Didn't retrieve data .Didn't retrieve data . /1341242970Didn't retrieve data .Didn't retrieve data . /1341242967Didn't retrieve data .Didn't retrieve data . /1341242380Didn't retrieve data .Didn't retrieve data . /1341242375Didn't retrieve data .Didn't retrieve data . /1341241949Didn't retrieve data .Didn't retrieve data . /1341241752Didn't retrieve data .Didn't retrieve data . /1341241599Didn't retrieve data .Didn't retrieve data . /1341241565Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341243040', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242973', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242970', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242967', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242380', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242375', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241949', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241752', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241599', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241565', None, None, None, None, None, '2646475') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013481378555297852 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10000395774841309 time spend to save output : 0.013987541198730469 total time spend for step 4 : 0.11399149894714355 step5:rle_unique_nms_with_priority Fri Mar 7 00:17:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 19 nb_hashtags : 5 time to prepare the origin masks : 15.241255521774292 time for calcul the mask position with numpy : 0.41106677055358887 nb_pixel_total : 5250770 time to create 1 rle with new method : 0.7634923458099365 time for calcul the mask position with numpy : 0.039720773696899414 nb_pixel_total : 12602 time to create 1 rle with old method : 0.0305788516998291 time for calcul the mask position with numpy : 0.07593107223510742 nb_pixel_total : 51510 time to create 1 rle with old method : 0.06086897850036621 time for calcul the mask position with numpy : 0.04581093788146973 nb_pixel_total : 98919 time to create 1 rle with old method : 0.11769628524780273 time for calcul the mask position with numpy : 0.03815293312072754 nb_pixel_total : 11426 time to create 1 rle with old method : 0.018607616424560547 time for calcul the mask position with numpy : 0.03412365913391113 nb_pixel_total : 13436 time to create 1 rle with old method : 0.015510082244873047 time for calcul the mask position with numpy : 0.030132770538330078 nb_pixel_total : 3123 time to create 1 rle with old method : 0.0036313533782958984 time for calcul the mask position with numpy : 0.025403261184692383 nb_pixel_total : 44622 time to create 1 rle with old method : 0.05760002136230469 time for calcul the mask position with numpy : 0.023068904876708984 nb_pixel_total : 7950 time to create 1 rle with old method : 0.008823871612548828 time for calcul the mask position with numpy : 0.023696422576904297 nb_pixel_total : 40783 time to create 1 rle with old method : 0.04767584800720215 time for calcul the mask position with numpy : 0.024789810180664062 nb_pixel_total : 117920 time to create 1 rle with old method : 0.1450517177581787 time for calcul the mask position with numpy : 0.02690434455871582 nb_pixel_total : 127250 time to create 1 rle with old method : 0.1457076072692871 time for calcul the mask position with numpy : 0.02410745620727539 nb_pixel_total : 12098 time to create 1 rle with old method : 0.015455007553100586 time for calcul the mask position with numpy : 0.029491424560546875 nb_pixel_total : 592338 time to create 1 rle with new method : 0.6264729499816895 time for calcul the mask position with numpy : 0.024848461151123047 nb_pixel_total : 26304 time to create 1 rle with old method : 0.031168699264526367 time for calcul the mask position with numpy : 0.03143501281738281 nb_pixel_total : 211014 time to create 1 rle with new method : 0.6860196590423584 time for calcul the mask position with numpy : 0.02289581298828125 nb_pixel_total : 55742 time to create 1 rle with old method : 0.06331110000610352 time for calcul the mask position with numpy : 0.023912668228149414 nb_pixel_total : 202532 time to create 1 rle with new method : 0.7868127822875977 time for calcul the mask position with numpy : 0.022822856903076172 nb_pixel_total : 133911 time to create 1 rle with old method : 0.1482532024383545 time for calcul the mask position with numpy : 0.023260831832885742 nb_pixel_total : 35990 time to create 1 rle with old method : 0.0393221378326416 create new chi : 4.92494797706604 time to delete rle : 0.014668941497802734 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 15092 TO DO : save crop sub photo not yet done ! save time : 0.8990459442138672 nb_obj : 1 nb_hashtags : 1 time to prepare the origin masks : 5.545318365097046 time for calcul the mask position with numpy : 0.04122138023376465 nb_pixel_total : 1525915 time to create 1 rle with new method : 1.2153129577636719 time for calcul the mask position with numpy : 1.0567777156829834 nb_pixel_total : 5524325 time to create 1 rle with new method : 1.0640089511871338 create new chi : 3.4441380500793457 time to delete rle : 0.0009493827819824219 batch 1 Loaded 2 chid ids of type : 3726 Number RLEs to save : 7888 TO DO : save crop sub photo not yet done ! save time : 0.5116047859191895 nb_obj : 16 nb_hashtags : 6 time to prepare the origin masks : 21.521921396255493 time for calcul the mask position with numpy : 0.9754157066345215 nb_pixel_total : 5852357 time to create 1 rle with new method : 0.7542991638183594 time for calcul the mask position with numpy : 0.023178577423095703 nb_pixel_total : 3543 time to create 1 rle with old method : 0.004506587982177734 time for calcul the mask position with numpy : 0.03693723678588867 nb_pixel_total : 2630 time to create 1 rle with old method : 0.004633665084838867 time for calcul the mask position with numpy : 0.04832291603088379 nb_pixel_total : 11587 time to create 1 rle with old method : 0.017786026000976562 time for calcul the mask position with numpy : 0.039754390716552734 nb_pixel_total : 6032 time to create 1 rle with old method : 0.006702899932861328 time for calcul the mask position with numpy : 0.04344892501831055 nb_pixel_total : 78904 time to create 1 rle with old method : 0.08952522277832031 time for calcul the mask position with numpy : 0.044684648513793945 nb_pixel_total : 61273 time to create 1 rle with old method : 0.06905484199523926 time for calcul the mask position with numpy : 0.03961992263793945 nb_pixel_total : 2546 time to create 1 rle with old method : 0.003058910369873047 time for calcul the mask position with numpy : 0.03446531295776367 nb_pixel_total : 1082 time to create 1 rle with old method : 0.0013687610626220703 time for calcul the mask position with numpy : 0.03848457336425781 nb_pixel_total : 467238 time to create 1 rle with new method : 1.126826286315918 time for calcul the mask position with numpy : 0.02674388885498047 nb_pixel_total : 17447 time to create 1 rle with old method : 0.020405292510986328 time for calcul the mask position with numpy : 0.024349689483642578 nb_pixel_total : 247729 time to create 1 rle with new method : 1.1791698932647705 time for calcul the mask position with numpy : 0.022382259368896484 nb_pixel_total : 1072 time to create 1 rle with old method : 0.0014410018920898438 time for calcul the mask position with numpy : 0.022578001022338867 nb_pixel_total : 77605 time to create 1 rle with old method : 0.08762359619140625 time for calcul the mask position with numpy : 0.023177623748779297 nb_pixel_total : 63976 time to create 1 rle with old method : 0.0710759162902832 time for calcul the mask position with numpy : 0.023432016372680664 nb_pixel_total : 97251 time to create 1 rle with old method : 0.11168551445007324 time for calcul the mask position with numpy : 0.02612018585205078 nb_pixel_total : 57968 time to create 1 rle with old method : 0.07232880592346191 create new chi : 5.217852592468262 time to delete rle : 0.002032756805419922 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 11658 TO DO : save crop sub photo not yet done ! save time : 0.6939949989318848 nb_obj : 18 nb_hashtags : 8 time to prepare the origin masks : 19.30861520767212 time for calcul the mask position with numpy : 0.29680800437927246 nb_pixel_total : 5690667 time to create 1 rle with new method : 1.2572507858276367 time for calcul the mask position with numpy : 0.03220534324645996 nb_pixel_total : 45 time to create 1 rle with old method : 0.0005736351013183594 time for calcul the mask position with numpy : 0.03280329704284668 nb_pixel_total : 46630 time to create 1 rle with old method : 0.05246710777282715 time for calcul the mask position with numpy : 0.033632755279541016 nb_pixel_total : 29736 time to create 1 rle with old method : 0.035324811935424805 time for calcul the mask position with numpy : 0.04749107360839844 nb_pixel_total : 116981 time to create 1 rle with old method : 0.13649201393127441 time for calcul the mask position with numpy : 0.039510488510131836 nb_pixel_total : 15517 time to create 1 rle with old method : 0.017302989959716797 time for calcul the mask position with numpy : 0.03554201126098633 nb_pixel_total : 116576 time to create 1 rle with old method : 0.13225054740905762 time for calcul the mask position with numpy : 0.03588747978210449 nb_pixel_total : 112367 time to create 1 rle with old method : 0.1307358741760254 time for calcul the mask position with numpy : 0.042403459548950195 nb_pixel_total : 197641 time to create 1 rle with new method : 1.6562557220458984 time for calcul the mask position with numpy : 0.031961917877197266 nb_pixel_total : 3722 time to create 1 rle with old method : 0.004014015197753906 time for calcul the mask position with numpy : 0.03144073486328125 nb_pixel_total : 29116 time to create 1 rle with old method : 0.03107738494873047 time for calcul the mask position with numpy : 0.03140592575073242 nb_pixel_total : 84850 time to create 1 rle with old method : 0.09018611907958984 time for calcul the mask position with numpy : 0.03050065040588379 nb_pixel_total : 1562 time to create 1 rle with old method : 0.0018954277038574219 time for calcul the mask position with numpy : 0.030073165893554688 nb_pixel_total : 116816 time to create 1 rle with old method : 0.1243596076965332 time for calcul the mask position with numpy : 0.020650625228881836 nb_pixel_total : 74746 time to create 1 rle with old method : 0.07735395431518555 time for calcul the mask position with numpy : 0.019987821578979492 nb_pixel_total : 82298 time to create 1 rle with old method : 0.08653926849365234 time for calcul the mask position with numpy : 0.021553516387939453 nb_pixel_total : 253585 time to create 1 rle with new method : 0.6041109561920166 time for calcul the mask position with numpy : 0.020755290985107422 nb_pixel_total : 52043 time to create 1 rle with old method : 0.05362439155578613 time for calcul the mask position with numpy : 0.020500659942626953 nb_pixel_total : 25342 time to create 1 rle with old method : 0.028885602951049805 create new chi : 5.457859754562378 time to delete rle : 0.001332998275756836 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 15453 TO DO : save crop sub photo not yet done ! save time : 0.8857674598693848 nb_obj : 27 nb_hashtags : 7 time to prepare the origin masks : 10.799235820770264 time for calcul the mask position with numpy : 4.144731521606445 nb_pixel_total : 5793429 time to create 1 rle with new method : 2.8512704372406006 time for calcul the mask position with numpy : 0.02980780601501465 nb_pixel_total : 38449 time to create 1 rle with old method : 0.04192209243774414 time for calcul the mask position with numpy : 0.02911090850830078 nb_pixel_total : 29159 time to create 1 rle with old method : 0.03413200378417969 time for calcul the mask position with numpy : 0.02875375747680664 nb_pixel_total : 26834 time to create 1 rle with old method : 0.03297829627990723 time for calcul the mask position with numpy : 0.02939295768737793 nb_pixel_total : 74049 time to create 1 rle with old method : 0.08450651168823242 time for calcul the mask position with numpy : 0.029202699661254883 nb_pixel_total : 12719 time to create 1 rle with old method : 0.013911962509155273 time for calcul the mask position with numpy : 0.028079986572265625 nb_pixel_total : 42349 time to create 1 rle with old method : 0.04669594764709473 time for calcul the mask position with numpy : 0.028579235076904297 nb_pixel_total : 4909 time to create 1 rle with old method : 0.005726814270019531 time for calcul the mask position with numpy : 0.02829122543334961 nb_pixel_total : 42787 time to create 1 rle with old method : 0.061896562576293945 time for calcul the mask position with numpy : 0.03898000717163086 nb_pixel_total : 94705 time to create 1 rle with old method : 0.11215734481811523 time for calcul the mask position with numpy : 0.029569625854492188 nb_pixel_total : 17952 time to create 1 rle with old method : 0.020125150680541992 time for calcul the mask position with numpy : 0.029796838760375977 nb_pixel_total : 116418 time to create 1 rle with old method : 0.12807655334472656 time for calcul the mask position with numpy : 0.02915644645690918 nb_pixel_total : 56715 time to create 1 rle with old method : 0.06255626678466797 time for calcul the mask position with numpy : 0.029026508331298828 nb_pixel_total : 435 time to create 1 rle with old method : 0.0007822513580322266 time for calcul the mask position with numpy : 0.03213381767272949 nb_pixel_total : 215028 time to create 1 rle with new method : 1.0342364311218262 time for calcul the mask position with numpy : 0.029910564422607422 nb_pixel_total : 41591 time to create 1 rle with old method : 0.04693603515625 time for calcul the mask position with numpy : 0.029432296752929688 nb_pixel_total : 29916 time to create 1 rle with old method : 0.03698420524597168 time for calcul the mask position with numpy : 0.029161691665649414 nb_pixel_total : 1591 time to create 1 rle with old method : 0.0022735595703125 time for calcul the mask position with numpy : 0.031394243240356445 nb_pixel_total : 48503 time to create 1 rle with old method : 0.06428003311157227 time for calcul the mask position with numpy : 0.02950763702392578 nb_pixel_total : 152997 time to create 1 rle with new method : 1.7420375347137451 time for calcul the mask position with numpy : 0.02895665168762207 nb_pixel_total : 40956 time to create 1 rle with old method : 0.045232295989990234 time for calcul the mask position with numpy : 0.02902507781982422 nb_pixel_total : 34516 time to create 1 rle with old method : 0.045122623443603516 time for calcul the mask position with numpy : 0.02986454963684082 nb_pixel_total : 34931 time to create 1 rle with old method : 0.03924202919006348 time for calcul the mask position with numpy : 0.028956174850463867 nb_pixel_total : 14566 time to create 1 rle with old method : 0.016194581985473633 time for calcul the mask position with numpy : 0.02873086929321289 nb_pixel_total : 12040 time to create 1 rle with old method : 0.013511180877685547 time for calcul the mask position with numpy : 0.029565811157226562 nb_pixel_total : 19501 time to create 1 rle with old method : 0.022470712661743164 time for calcul the mask position with numpy : 0.028739213943481445 nb_pixel_total : 51142 time to create 1 rle with old method : 0.056693077087402344 time for calcul the mask position with numpy : 0.028443336486816406 nb_pixel_total : 2053 time to create 1 rle with old method : 0.002343893051147461 create new chi : 11.690949440002441 time to delete rle : 0.0016679763793945312 batch 1 Loaded 28 chid ids of type : 3726 Number RLEs to save : 18005 TO DO : save crop sub photo not yet done ! save time : 1.366471767425537 nb_obj : 18 nb_hashtags : 7 time to prepare the origin masks : 22.882022619247437 time for calcul the mask position with numpy : 0.9481284618377686 nb_pixel_total : 5077925 time to create 1 rle with new method : 1.2583048343658447 time for calcul the mask position with numpy : 0.044274091720581055 nb_pixel_total : 8436 time to create 1 rle with old method : 0.01170039176940918 time for calcul the mask position with numpy : 0.04594779014587402 nb_pixel_total : 103055 time to create 1 rle with old method : 0.11330795288085938 time for calcul the mask position with numpy : 0.038893938064575195 nb_pixel_total : 123988 time to create 1 rle with old method : 0.1615133285522461 time for calcul the mask position with numpy : 0.03520321846008301 nb_pixel_total : 166347 time to create 1 rle with new method : 0.9114029407501221 time for calcul the mask position with numpy : 0.03363513946533203 nb_pixel_total : 45214 time to create 1 rle with old method : 0.049012184143066406 time for calcul the mask position with numpy : 0.03721117973327637 nb_pixel_total : 4539 time to create 1 rle with old method : 0.005151510238647461 time for calcul the mask position with numpy : 0.03406381607055664 nb_pixel_total : 7211 time to create 1 rle with old method : 0.007967710494995117 time for calcul the mask position with numpy : 0.03597569465637207 nb_pixel_total : 27131 time to create 1 rle with old method : 0.03363513946533203 time for calcul the mask position with numpy : 0.034620046615600586 nb_pixel_total : 57432 time to create 1 rle with old method : 0.06241464614868164 time for calcul the mask position with numpy : 0.03585505485534668 nb_pixel_total : 135140 time to create 1 rle with old method : 0.17330551147460938 time for calcul the mask position with numpy : 0.033143043518066406 nb_pixel_total : 1633 time to create 1 rle with old method : 0.0018815994262695312 time for calcul the mask position with numpy : 0.03448176383972168 nb_pixel_total : 10896 time to create 1 rle with old method : 0.012134552001953125 time for calcul the mask position with numpy : 0.034903764724731445 nb_pixel_total : 64466 time to create 1 rle with old method : 0.0716855525970459 time for calcul the mask position with numpy : 0.03528594970703125 nb_pixel_total : 154136 time to create 1 rle with new method : 0.7710659503936768 time for calcul the mask position with numpy : 0.025164365768432617 nb_pixel_total : 25136 time to create 1 rle with old method : 0.04018807411193848 time for calcul the mask position with numpy : 0.024144887924194336 nb_pixel_total : 4051 time to create 1 rle with old method : 0.004590034484863281 time for calcul the mask position with numpy : 0.03593754768371582 nb_pixel_total : 993825 time to create 1 rle with new method : 2.15543270111084 time for calcul the mask position with numpy : 0.04354500770568848 nb_pixel_total : 39679 time to create 1 rle with old method : 0.0452876091003418 create new chi : 7.597158908843994 time to delete rle : 0.0014522075653076172 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 15272 TO DO : save crop sub photo not yet done ! save time : 0.9009528160095215 nb_obj : 25 nb_hashtags : 7 time to prepare the origin masks : 28.807852506637573 time for calcul the mask position with numpy : 1.2798807621002197 nb_pixel_total : 5230899 time to create 1 rle with new method : 0.9623692035675049 time for calcul the mask position with numpy : 0.03413128852844238 nb_pixel_total : 10946 time to create 1 rle with old method : 0.011809110641479492 time for calcul the mask position with numpy : 0.034456491470336914 nb_pixel_total : 22188 time to create 1 rle with old method : 0.028481483459472656 time for calcul the mask position with numpy : 0.04188847541809082 nb_pixel_total : 33861 time to create 1 rle with old method : 0.03704500198364258 time for calcul the mask position with numpy : 0.03670811653137207 nb_pixel_total : 152643 time to create 1 rle with new method : 1.1332850456237793 time for calcul the mask position with numpy : 0.03745222091674805 nb_pixel_total : 33105 time to create 1 rle with old method : 0.058656930923461914 time for calcul the mask position with numpy : 0.04421281814575195 nb_pixel_total : 71635 time to create 1 rle with old method : 0.08519101142883301 time for calcul the mask position with numpy : 0.046339988708496094 nb_pixel_total : 199857 time to create 1 rle with new method : 1.3736882209777832 time for calcul the mask position with numpy : 0.03327059745788574 nb_pixel_total : 9396 time to create 1 rle with old method : 0.009924650192260742 time for calcul the mask position with numpy : 0.03703165054321289 nb_pixel_total : 426849 time to create 1 rle with new method : 0.47928357124328613 time for calcul the mask position with numpy : 0.03675961494445801 nb_pixel_total : 22372 time to create 1 rle with old method : 0.02485036849975586 time for calcul the mask position with numpy : 0.036721229553222656 nb_pixel_total : 9421 time to create 1 rle with old method : 0.011094331741333008 time for calcul the mask position with numpy : 0.04613637924194336 nb_pixel_total : 6449 time to create 1 rle with old method : 0.00787973403930664 time for calcul the mask position with numpy : 0.03882026672363281 nb_pixel_total : 174133 time to create 1 rle with new method : 0.9924309253692627 time for calcul the mask position with numpy : 0.036226511001586914 nb_pixel_total : 45911 time to create 1 rle with old method : 0.05010700225830078 time for calcul the mask position with numpy : 0.03547263145446777 nb_pixel_total : 5598 time to create 1 rle with old method : 0.006181001663208008 time for calcul the mask position with numpy : 0.034670114517211914 nb_pixel_total : 43264 time to create 1 rle with old method : 0.04721426963806152 time for calcul the mask position with numpy : 0.0346522331237793 nb_pixel_total : 6676 time to create 1 rle with old method : 0.007369279861450195 time for calcul the mask position with numpy : 0.028438091278076172 nb_pixel_total : 57589 time to create 1 rle with old method : 0.06265687942504883 time for calcul the mask position with numpy : 0.03829669952392578 nb_pixel_total : 3095 time to create 1 rle with old method : 0.0038962364196777344 time for calcul the mask position with numpy : 0.03411412239074707 nb_pixel_total : 41831 time to create 1 rle with old method : 0.044939517974853516 time for calcul the mask position with numpy : 0.03447079658508301 nb_pixel_total : 85640 time to create 1 rle with old method : 0.08942866325378418 time for calcul the mask position with numpy : 0.03631305694580078 nb_pixel_total : 318209 time to create 1 rle with new method : 0.9024040699005127 time for calcul the mask position with numpy : 0.03413200378417969 nb_pixel_total : 27654 time to create 1 rle with old method : 0.03052043914794922 time for calcul the mask position with numpy : 0.03315091133117676 nb_pixel_total : 6979 time to create 1 rle with old method : 0.007696390151977539 time for calcul the mask position with numpy : 0.03412580490112305 nb_pixel_total : 4040 time to create 1 rle with old method : 0.0043544769287109375 create new chi : 8.8369300365448 time to delete rle : 0.0024194717407226562 batch 1 Loaded 26 chid ids of type : 3726 Number RLEs to save : 19046 TO DO : save crop sub photo not yet done ! save time : 1.1266093254089355 nb_obj : 18 nb_hashtags : 6 time to prepare the origin masks : 25.312379837036133 time for calcul the mask position with numpy : 0.5376620292663574 nb_pixel_total : 5843973 time to create 1 rle with new method : 0.855196475982666 time for calcul the mask position with numpy : 0.04374837875366211 nb_pixel_total : 126523 time to create 1 rle with old method : 0.14408087730407715 time for calcul the mask position with numpy : 0.04655909538269043 nb_pixel_total : 31794 time to create 1 rle with old method : 0.0372319221496582 time for calcul the mask position with numpy : 0.044020652770996094 nb_pixel_total : 81392 time to create 1 rle with old method : 0.1125648021697998 time for calcul the mask position with numpy : 0.04089021682739258 nb_pixel_total : 25697 time to create 1 rle with old method : 0.02898693084716797 time for calcul the mask position with numpy : 0.04273414611816406 nb_pixel_total : 15949 time to create 1 rle with old method : 0.018368959426879883 time for calcul the mask position with numpy : 0.041814327239990234 nb_pixel_total : 39308 time to create 1 rle with old method : 0.04407095909118652 time for calcul the mask position with numpy : 0.03943157196044922 nb_pixel_total : 61299 time to create 1 rle with old method : 0.08071517944335938 time for calcul the mask position with numpy : 0.0416719913482666 nb_pixel_total : 15590 time to create 1 rle with old method : 0.018152952194213867 time for calcul the mask position with numpy : 0.03680586814880371 nb_pixel_total : 19437 time to create 1 rle with old method : 0.024514436721801758 time for calcul the mask position with numpy : 0.023339033126831055 nb_pixel_total : 2364 time to create 1 rle with old method : 0.003210306167602539 time for calcul the mask position with numpy : 0.02307748794555664 nb_pixel_total : 145990 time to create 1 rle with old method : 0.17165803909301758 time for calcul the mask position with numpy : 0.024967193603515625 nb_pixel_total : 186225 time to create 1 rle with new method : 0.7452049255371094 time for calcul the mask position with numpy : 0.026382923126220703 nb_pixel_total : 4410 time to create 1 rle with old method : 0.00727534294128418 time for calcul the mask position with numpy : 0.025000333786010742 nb_pixel_total : 1765 time to create 1 rle with old method : 0.0029268264770507812 time for calcul the mask position with numpy : 0.0282595157623291 nb_pixel_total : 200019 time to create 1 rle with new method : 0.9372973442077637 time for calcul the mask position with numpy : 0.0259244441986084 nb_pixel_total : 123574 time to create 1 rle with old method : 0.1417379379272461 time for calcul the mask position with numpy : 0.04416775703430176 nb_pixel_total : 104422 time to create 1 rle with old method : 0.1638939380645752 time for calcul the mask position with numpy : 0.05378460884094238 nb_pixel_total : 20509 time to create 1 rle with old method : 0.03639030456542969 create new chi : 4.858438730239868 time to delete rle : 0.010472774505615234 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 13503 TO DO : save crop sub photo not yet done ! save time : 1.0201478004455566 nb_obj : 4 nb_hashtags : 3 time to prepare the origin masks : 6.165332317352295 time for calcul the mask position with numpy : 0.06293129920959473 nb_pixel_total : 2543984 time to create 1 rle with new method : 0.9302809238433838 time for calcul the mask position with numpy : 0.024940013885498047 nb_pixel_total : 6748 time to create 1 rle with old method : 0.007688999176025391 time for calcul the mask position with numpy : 0.059233665466308594 nb_pixel_total : 559672 time to create 1 rle with new method : 0.7108006477355957 time for calcul the mask position with numpy : 0.2264420986175537 nb_pixel_total : 3888359 time to create 1 rle with new method : 1.4707293510437012 time for calcul the mask position with numpy : 0.028314828872680664 nb_pixel_total : 51477 time to create 1 rle with old method : 0.08151865005493164 create new chi : 3.6885881423950195 time to delete rle : 0.0011632442474365234 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 12203 TO DO : save crop sub photo not yet done ! save time : 0.763007402420044 nb_obj : 10 nb_hashtags : 5 time to prepare the origin masks : 10.794785737991333 time for calcul the mask position with numpy : 0.8524191379547119 nb_pixel_total : 6464448 time to create 1 rle with new method : 0.8538498878479004 time for calcul the mask position with numpy : 0.03746390342712402 nb_pixel_total : 14407 time to create 1 rle with old method : 0.019345998764038086 time for calcul the mask position with numpy : 0.034456491470336914 nb_pixel_total : 197369 time to create 1 rle with new method : 1.0492243766784668 time for calcul the mask position with numpy : 0.04170417785644531 nb_pixel_total : 23374 time to create 1 rle with old method : 0.03313875198364258 time for calcul the mask position with numpy : 0.03652143478393555 nb_pixel_total : 70746 time to create 1 rle with old method : 0.07970857620239258 time for calcul the mask position with numpy : 0.025401592254638672 nb_pixel_total : 6746 time to create 1 rle with old method : 0.009328842163085938 time for calcul the mask position with numpy : 0.03172874450683594 nb_pixel_total : 49960 time to create 1 rle with old method : 0.06277823448181152 time for calcul the mask position with numpy : 0.03053736686706543 nb_pixel_total : 42959 time to create 1 rle with old method : 0.05142474174499512 time for calcul the mask position with numpy : 0.02707195281982422 nb_pixel_total : 83771 time to create 1 rle with old method : 0.1038665771484375 time for calcul the mask position with numpy : 0.026848554611206055 nb_pixel_total : 78630 time to create 1 rle with old method : 0.10910367965698242 time for calcul the mask position with numpy : 0.02557063102722168 nb_pixel_total : 17830 time to create 1 rle with old method : 0.02066779136657715 create new chi : 3.6415791511535645 time to delete rle : 0.0009903907775878906 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 8361 TO DO : save crop sub photo not yet done ! save time : 0.6071555614471436 map_output_result : {1341243040: (0.0, 'Should be the crop_list due to order', 0), 1341242973: (0.0, 'Should be the crop_list due to order', 0), 1341242970: (0.0, 'Should be the crop_list due to order', 0), 1341242967: (0.0, 'Should be the crop_list due to order', 0), 1341242380: (0.0, 'Should be the crop_list due to order', 0), 1341242375: (0.0, 'Should be the crop_list due to order', 0), 1341241949: (0.0, 'Should be the crop_list due to order', 0), 1341241752: (0.0, 'Should be the crop_list due to order', 0), 1341241599: (0.0, 'Should be the crop_list due to order', 0), 1341241565: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1341243040, 1341242973, 1341242970, 1341242967, 1341242380, 1341242375, 1341241949, 1341241752, 1341241599, 1341241565] Looping around the photos to save general results len do output : 10 /1341243040.Didn't retrieve data . /1341242973.Didn't retrieve data . /1341242970.Didn't retrieve data . /1341242967.Didn't retrieve data . /1341242380.Didn't retrieve data . /1341242375.Didn't retrieve data . /1341241949.Didn't retrieve data . /1341241752.Didn't retrieve data . /1341241599.Didn't retrieve data . /1341241565.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341243040', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242973', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242970', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242967', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242380', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242375', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241949', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241752', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241599', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241565', None, None, None, None, None, '2646475') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014720678329467773 save_final save missing photos in datou_result : time spend for datou_step_exec : 235.85069274902344 time spend to save output : 0.025138378143310547 total time spend for step 5 : 235.87583112716675 step6:crop_condition Fri Mar 7 00:21:13 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 166 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 4869462 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741303278_121508 we have uploaded 9 photos in the portfolio 4869462 time of upload the photos Elapsed time : 3.1291000843048096 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 15 About to insert : list_path_to_insert length 15 new photo from crops ! About to upload 15 photos upload in portfolio : 4869462 init cache_photo without model_param we have 15 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741303291_121508 we have uploaded 15 photos in the portfolio 4869462 time of upload the photos Elapsed time : 5.086543321609497 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741303299_121508 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.8790364265441895 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741303305_121508 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.1631531715393066 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741303310_121508 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.6644384860992432 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741303316_121508 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.8445210456848145 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1341243040, 1341242973, 1341242970, 1341242967, 1341242380, 1341242375, 1341241949, 1341241752, 1341241599, 1341241565] Looping around the photos to save general results len do output : 41 /1341338906Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338907Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338908Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338909Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338910Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338911Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338912Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338913Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338914Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338962Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338963Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338964Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338965Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338966Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338967Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338968Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338969Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338970Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338971Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338972Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338973Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338974Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338975Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341338976Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339015Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339020Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339024Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339028Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339169Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339173Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339176Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339180Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339184Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339188Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339192Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339206Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339207Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339208Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339216Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339217Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341339218Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341243040', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242973', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242970', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242967', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242380', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242375', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241949', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241752', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241599', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241565', None, None, None, None, None, '2646475') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 133 time used for this insertion : 0.03661203384399414 save_final save missing photos in datou_result : time spend for datou_step_exec : 45.99855017662048 time spend to save output : 0.047448158264160156 total time spend for step 6 : 46.045998334884644 step7:ventilate_hashtags_in_portfolio Fri Mar 7 00:21:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 21131106 get user id for portfolio 21131106 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131106 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('plastique','autre_refus','mal_croppe','Carton_brun','metal','environnement','Carton_gris','Teint_Dans_La_Masse','cartonnette','kraft','papier','flou')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131106 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('plastique','autre_refus','mal_croppe','Carton_brun','metal','environnement','Carton_gris','Teint_Dans_La_Masse','cartonnette','kraft','papier','flou')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131106 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('plastique','autre_refus','mal_croppe','Carton_brun','metal','environnement','Carton_gris','Teint_Dans_La_Masse','cartonnette','kraft','papier','flou')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/21131264,21131265,21131266,21131267,21131268,21131269,21131270,21131271,21131272,21131273,21131274,21131275?tags=plastique,autre_refus,mal_croppe,Carton_brun,metal,environnement,Carton_gris,Teint_Dans_La_Masse,cartonnette,kraft,papier,flou Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1341243040, 1341242973, 1341242970, 1341242967, 1341242380, 1341242375, 1341241949, 1341241752, 1341241599, 1341241565] Looping around the photos to save general results len do output : 1 /21131106. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341243040', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242973', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242970', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242967', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242380', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242375', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241949', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241752', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241599', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241565', None, None, None, None, None, '2646475') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.015471696853637695 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.6807963848114014 time spend to save output : 0.01577901840209961 total time spend for step 7 : 1.696575403213501 step8:final Fri Mar 7 00:22:01 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1341243040: ('0.1208628538400596',), 1341242973: ('0.1208628538400596',), 1341242970: ('0.1208628538400596',), 1341242967: ('0.1208628538400596',), 1341242380: ('0.1208628538400596',), 1341242375: ('0.1208628538400596',), 1341241949: ('0.1208628538400596',), 1341241752: ('0.1208628538400596',), 1341241599: ('0.1208628538400596',), 1341241565: ('0.1208628538400596',)} new output for save of step final : {1341243040: ('0.1208628538400596',), 1341242973: ('0.1208628538400596',), 1341242970: ('0.1208628538400596',), 1341242967: ('0.1208628538400596',), 1341242380: ('0.1208628538400596',), 1341242375: ('0.1208628538400596',), 1341241949: ('0.1208628538400596',), 1341241752: ('0.1208628538400596',), 1341241599: ('0.1208628538400596',), 1341241565: ('0.1208628538400596',)} [1341243040, 1341242973, 1341242970, 1341242967, 1341242380, 1341242375, 1341241949, 1341241752, 1341241599, 1341241565] Looping around the photos to save general results len do output : 10 /1341243040.Didn't retrieve data . /1341242973.Didn't retrieve data . /1341242970.Didn't retrieve data . /1341242967.Didn't retrieve data . /1341242380.Didn't retrieve data . /1341242375.Didn't retrieve data . /1341241949.Didn't retrieve data . /1341241752.Didn't retrieve data . /1341241599.Didn't retrieve data . /1341241565.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341243040', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242973', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242970', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242967', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242380', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242375', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241949', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241752', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241599', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241565', None, None, None, None, None, '2646475') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013456583023071289 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10997986793518066 time spend to save output : 0.014051437377929688 total time spend for step 8 : 0.12403130531311035 step9:velours_tree Fri Mar 7 00:22:01 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.10120820999145508 time spend to save output : 5.14984130859375e-05 total time spend for step 9 : 0.10125970840454102 step10:send_mail_cod Fri Mar 7 00:22:01 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21131106_07-03-2025_00_22_01.pdf 21131264 imagette211312641741303321 21131265 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211312651741303321 21131266 imagette211312661741303322 21131268 imagette211312681741303322 21131272 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211312721741303322 21131273 change filename to text .change filename to text .change filename to text .imagette211312731741303323 21131275 imagette211312751741303323 21131267 change filename to text .change filename to text .change filename to text .imagette211312671741303323 21131270 change filename to text .change filename to text .change filename to text .change filename to text .imagette211312701741303324 21131271 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211312711741303324 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21131106 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21131264,21131265,21131266,21131267,21131268,21131269,21131270,21131271,21131272,21131273,21131274,21131275?tags=plastique,autre_refus,mal_croppe,Carton_brun,metal,environnement,Carton_gris,Teint_Dans_La_Masse,cartonnette,kraft,papier,flou your option no_mail is active, we will not send the real mail to your client args[1341243040] : ('0.1208628538400596',) no score found for photo 1341243040 We are sending mail with results at report@fotonower.com args[1341242973] : ('0.1208628538400596',) no score found for photo 1341242973 We are sending mail with results at report@fotonower.com args[1341242970] : ('0.1208628538400596',) no score found for photo 1341242970 We are sending mail with results at report@fotonower.com args[1341242967] : ('0.1208628538400596',) no score found for photo 1341242967 We are sending mail with results at report@fotonower.com args[1341242380] : ('0.1208628538400596',) no score found for photo 1341242380 We are sending mail with results at report@fotonower.com args[1341242375] : ('0.1208628538400596',) no score found for photo 1341242375 We are sending mail with results at report@fotonower.com args[1341241949] : ('0.1208628538400596',) no score found for photo 1341241949 We are sending mail with results at report@fotonower.com args[1341241752] : ('0.1208628538400596',) no score found for photo 1341241752 We are sending mail with results at report@fotonower.com args[1341241599] : ('0.1208628538400596',) no score found for photo 1341241599 We are sending mail with results at report@fotonower.com args[1341241565] : ('0.1208628538400596',) no score found for photo 1341241565 We are sending mail with results at report@fotonower.com refus_total : 0.1208628538400596 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=21131106 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1341241560,1341243040,1341242967,1341242380,1341242375,1341241949,1341241752,1341241475,1341241555,1341241552,1341241510,1341241504,1341241487,1341241483,1341241549,1341241565,1341241599,1341241547,1341242970,1341242973) Found this number of photos: 20 begin to download photo : 1341241560 begin to download photo : 1341241949 begin to download photo : 1341241510 begin to download photo : 1341241565 download finish for photo 1341241510 begin to download photo : 1341241504 download finish for photo 1341241565 begin to download photo : 1341241599 download finish for photo 1341241560 begin to download photo : 1341243040 download finish for photo 1341241949 begin to download photo : 1341241752 download finish for photo 1341241504 begin to download photo : 1341241487 download finish for photo 1341241599 begin to download photo : 1341241547 download finish for photo 1341243040 begin to download photo : 1341242967 download finish for photo 1341241752 begin to download photo : 1341241475 download finish for photo 1341241547 begin to download photo : 1341242970 download finish for photo 1341241487 begin to download photo : 1341241483 download finish for photo 1341242967 begin to download photo : 1341242380 download finish for photo 1341241475 begin to download photo : 1341241555 download finish for photo 1341242380 begin to download photo : 1341242375 download finish for photo 1341242970 begin to download photo : 1341242973 download finish for photo 1341241483 begin to download photo : 1341241549 download finish for photo 1341241555 begin to download photo : 1341241552 download finish for photo 1341242375 download finish for photo 1341242973 download finish for photo 1341241552 download finish for photo 1341241549 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131106_07-03-2025_00_22_01.pdf results_Auto_P21131106_07-03-2025_00_22_01.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131106_07-03-2025_00_22_01.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','21131106','results_Auto_P21131106_07-03-2025_00_22_01.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131106_07-03-2025_00_22_01.pdf','pdf','','0.84','0.1208628538400596') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1341243040, 1341242973, 1341242970, 1341242967, 1341242380, 1341242375, 1341241949, 1341241752, 1341241599, 1341241565] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341243040', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242973', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242970', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242967', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242380', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242375', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241949', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241752', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241599', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241565', None, None, None, None, None, '2646475') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.013637542724609375 save_final save missing photos in datou_result : time spend for datou_step_exec : 9.102300643920898 time spend to save output : 0.01384878158569336 total time spend for step 10 : 9.116149425506592 step11:split_time_score Fri Mar 7 00:22:10 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('12', 21),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 06032025 21131106 Nombre de photos uploadées : 21 / 23040 (0%) 06032025 21131106 Nombre de photos taguées (types de déchets): 0 / 21 (0%) 06032025 21131106 Nombre de photos taguées (volume) : 0 / 21 (0%) elapsed_time : load_data_split_time_score 3.5762786865234375e-06 elapsed_time : order_list_meta_photo_and_scores 8.344650268554688e-06 ????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0015914440155029297 elapsed_time : insert_dashboard_record_day_entry 0.02425074577331543 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131086 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131087 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131088 order by id desc limit 1 Qualite : 0.08488825436881216 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21110324_06-03-2025_13_24_09.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21110324 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21110324 AND mptpi.`type`=3327 To do Qualite : 0.08456570303462264 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21110325_06-03-2025_09_27_50.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21110325 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21110325 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21110326 order by id desc limit 1 Qualite : 0.05850682993931322 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21110327_06-03-2025_09_32_24.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21110327 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21110327 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131089 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131090 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131091 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131092 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131093 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131094 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131095 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131096 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131097 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131098 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131099 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131100 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131101 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131102 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131103 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131104 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131105 order by id desc limit 1 Qualite : 0.1208628538400596 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131106_07-03-2025_00_22_01.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131106 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131106 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131107 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131108 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131109 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131110 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131111 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131112 order by id desc limit 1 Qualite : 0.09219659423566663 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131113_07-03-2025_00_21_45.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131113 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131113 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131114 order by id desc limit 1 Qualite : 0.0628226615711991 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131115_07-03-2025_00_19_15.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131115 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131115 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'06032025': {'nb_upload': 21, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1341243040, 1341242973, 1341242970, 1341242967, 1341242380, 1341242375, 1341241949, 1341241752, 1341241599, 1341241565] Looping around the photos to save general results len do output : 1 /21131106Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341243040', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242973', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242970', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242967', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242380', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341242375', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241949', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241752', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241599', None, None, None, None, None, '2646475') ('3459', None, None, None, None, None, None, None, '2646475') ('3459', '21131106', '1341241565', None, None, None, None, None, '2646475') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01628565788269043 save_final save missing photos in datou_result : time spend for datou_step_exec : 22.45469093322754 time spend to save output : 0.016587495803833008 total time spend for step 11 : 22.471278429031372 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 184.80user 263.62system 8:08.86elapsed 91%CPU (0avgtext+0avgdata 4418752maxresident)k 474680inputs+135944outputs (15818major+14514775minor)pagefaults 0swaps