python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 860455 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 15>10 please execute split_portfolio.py -i 21178362 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2652113'] with mtr_portfolio_ids : ['21178362'] and first list_photo_ids : [] new path : /proc/860455/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.68902850151062 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Sat Mar 8 14:52:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 8293 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-08 14:52:36.347814: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-08 14:52:36.379372: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-08 14:52:36.381234: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f9540000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-08 14:52:36.381268: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-08 14:52:36.386217: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-08 14:52:36.644774: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x29c58c80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-08 14:52:36.644824: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-08 14:52:36.646510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-08 14:52:36.648975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-08 14:52:36.677010: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-08 14:52:36.701108: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-08 14:52:36.704164: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-08 14:52:36.750811: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-08 14:52:36.756112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-08 14:52:36.809853: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-08 14:52:36.811533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-08 14:52:36.812032: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-08 14:52:36.812918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-08 14:52:36.812937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-08 14:52:36.812948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-08 14:52:36.815114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7653 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-08 14:52:37.265759: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-08 14:52:37.265874: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-08 14:52:37.265891: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-08 14:52:37.265907: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-08 14:52:37.265922: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-08 14:52:37.265937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-08 14:52:37.265951: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-08 14:52:37.265966: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-08 14:52:37.267186: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-08 14:52:37.268403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-08 14:52:37.268470: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-08 14:52:37.268488: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-08 14:52:37.268502: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-08 14:52:37.268516: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-08 14:52:37.268531: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-08 14:52:37.268544: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-08 14:52:37.268559: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-08 14:52:37.269725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-08 14:52:37.269760: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-08 14:52:37.269770: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-08 14:52:37.269778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-08 14:52:37.271037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7653 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-08 14:52:51.330375: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-08 14:52:51.696003: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 18 Detection mask done ! Trying to reset tf kernel 861031 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3978 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 9265 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.003876209259033203 nb_pixel_total : 174890 time to create 1 rle with new method : 0.009471654891967773 length of segment : 628 time for calcul the mask position with numpy : 1.281844139099121 nb_pixel_total : 1600998 time to create 1 rle with new method : 0.3205592632293701 length of segment : 2329 time for calcul the mask position with numpy : 0.00021982192993164062 nb_pixel_total : 8756 time to create 1 rle with old method : 0.010360479354858398 length of segment : 133 time for calcul the mask position with numpy : 0.0008792877197265625 nb_pixel_total : 45207 time to create 1 rle with old method : 0.06953239440917969 length of segment : 131 time for calcul the mask position with numpy : 0.006556510925292969 nb_pixel_total : 457358 time to create 1 rle with new method : 0.017038345336914062 length of segment : 994 time for calcul the mask position with numpy : 0.000476837158203125 nb_pixel_total : 21974 time to create 1 rle with old method : 0.026491641998291016 length of segment : 175 time for calcul the mask position with numpy : 0.00022411346435546875 nb_pixel_total : 13742 time to create 1 rle with old method : 0.016881227493286133 length of segment : 107 time for calcul the mask position with numpy : 0.0011827945709228516 nb_pixel_total : 80677 time to create 1 rle with old method : 0.1198430061340332 length of segment : 433 time for calcul the mask position with numpy : 0.0041272640228271484 nb_pixel_total : 251709 time to create 1 rle with new method : 0.013732194900512695 length of segment : 1230 time for calcul the mask position with numpy : 0.00048065185546875 nb_pixel_total : 19786 time to create 1 rle with old method : 0.023397207260131836 length of segment : 221 time for calcul the mask position with numpy : 0.0005574226379394531 nb_pixel_total : 23410 time to create 1 rle with old method : 0.030713558197021484 length of segment : 164 time for calcul the mask position with numpy : 0.003094911575317383 nb_pixel_total : 178849 time to create 1 rle with new method : 0.009077787399291992 length of segment : 597 time for calcul the mask position with numpy : 0.0019214153289794922 nb_pixel_total : 118505 time to create 1 rle with old method : 0.14085125923156738 length of segment : 431 time for calcul the mask position with numpy : 0.002101898193359375 nb_pixel_total : 127915 time to create 1 rle with old method : 0.15241098403930664 length of segment : 396 time for calcul the mask position with numpy : 0.0023458003997802734 nb_pixel_total : 152514 time to create 1 rle with new method : 0.005959510803222656 length of segment : 452 time for calcul the mask position with numpy : 0.002098560333251953 nb_pixel_total : 124839 time to create 1 rle with old method : 0.16084861755371094 length of segment : 363 time for calcul the mask position with numpy : 0.003015756607055664 nb_pixel_total : 179987 time to create 1 rle with new method : 0.007800102233886719 length of segment : 1565 time for calcul the mask position with numpy : 0.03110027313232422 nb_pixel_total : 1573744 time to create 1 rle with new method : 0.25430917739868164 length of segment : 2422 time for calcul the mask position with numpy : 0.0009989738464355469 nb_pixel_total : 24363 time to create 1 rle with old method : 0.029788970947265625 length of segment : 131 time for calcul the mask position with numpy : 0.0035054683685302734 nb_pixel_total : 117930 time to create 1 rle with old method : 0.13589835166931152 length of segment : 396 time for calcul the mask position with numpy : 0.010498046875 nb_pixel_total : 371225 time to create 1 rle with new method : 0.01934981346130371 length of segment : 1488 time for calcul the mask position with numpy : 0.001705169677734375 nb_pixel_total : 40367 time to create 1 rle with old method : 0.05087637901306152 length of segment : 173 time for calcul the mask position with numpy : 0.0006971359252929688 nb_pixel_total : 41439 time to create 1 rle with old method : 0.04978680610656738 length of segment : 236 time for calcul the mask position with numpy : 0.0017561912536621094 nb_pixel_total : 94025 time to create 1 rle with old method : 0.1083228588104248 length of segment : 409 time for calcul the mask position with numpy : 0.0010116100311279297 nb_pixel_total : 41450 time to create 1 rle with old method : 0.04908561706542969 length of segment : 161 time for calcul the mask position with numpy : 0.0014066696166992188 nb_pixel_total : 33289 time to create 1 rle with old method : 0.0399625301361084 length of segment : 187 time for calcul the mask position with numpy : 0.004651308059692383 nb_pixel_total : 142417 time to create 1 rle with old method : 0.18919873237609863 length of segment : 475 time for calcul the mask position with numpy : 0.0013866424560546875 nb_pixel_total : 21994 time to create 1 rle with old method : 0.03153848648071289 length of segment : 302 time for calcul the mask position with numpy : 0.0038056373596191406 nb_pixel_total : 177430 time to create 1 rle with new method : 0.007944107055664062 length of segment : 597 time for calcul the mask position with numpy : 0.26819896697998047 nb_pixel_total : 3527246 time to create 1 rle with new method : 1.0984816551208496 length of segment : 2585 time for calcul the mask position with numpy : 0.0006592273712158203 nb_pixel_total : 45555 time to create 1 rle with old method : 0.055670738220214844 length of segment : 187 time for calcul the mask position with numpy : 0.002242565155029297 nb_pixel_total : 130055 time to create 1 rle with old method : 0.1532280445098877 length of segment : 1258 time for calcul the mask position with numpy : 0.018910884857177734 nb_pixel_total : 1329527 time to create 1 rle with new method : 0.12345576286315918 length of segment : 2647 time for calcul the mask position with numpy : 0.0010960102081298828 nb_pixel_total : 54993 time to create 1 rle with old method : 0.0702812671661377 length of segment : 253 time for calcul the mask position with numpy : 0.0167238712310791 nb_pixel_total : 987982 time to create 1 rle with new method : 0.08621692657470703 length of segment : 1734 time for calcul the mask position with numpy : 0.013733625411987305 nb_pixel_total : 757523 time to create 1 rle with new method : 0.24344682693481445 length of segment : 1433 time for calcul the mask position with numpy : 0.0003955364227294922 nb_pixel_total : 11782 time to create 1 rle with old method : 0.015779495239257812 length of segment : 141 time for calcul the mask position with numpy : 0.002713441848754883 nb_pixel_total : 150112 time to create 1 rle with new method : 0.006580829620361328 length of segment : 577 time for calcul the mask position with numpy : 0.0013346672058105469 nb_pixel_total : 68376 time to create 1 rle with old method : 0.08368134498596191 length of segment : 423 time for calcul the mask position with numpy : 0.002382040023803711 nb_pixel_total : 95081 time to create 1 rle with old method : 0.11647891998291016 length of segment : 369 time for calcul the mask position with numpy : 0.00022077560424804688 nb_pixel_total : 7294 time to create 1 rle with old method : 0.018964052200317383 length of segment : 91 time for calcul the mask position with numpy : 0.016368389129638672 nb_pixel_total : 670068 time to create 1 rle with new method : 0.24945712089538574 length of segment : 1525 time for calcul the mask position with numpy : 0.0007994174957275391 nb_pixel_total : 15482 time to create 1 rle with old method : 0.027932405471801758 length of segment : 119 time for calcul the mask position with numpy : 0.0029180049896240234 nb_pixel_total : 74221 time to create 1 rle with old method : 0.08738899230957031 length of segment : 302 time for calcul the mask position with numpy : 0.0017294883728027344 nb_pixel_total : 36839 time to create 1 rle with old method : 0.04281282424926758 length of segment : 518 time for calcul the mask position with numpy : 0.002912759780883789 nb_pixel_total : 171121 time to create 1 rle with new method : 0.007117748260498047 length of segment : 577 time for calcul the mask position with numpy : 0.00028204917907714844 nb_pixel_total : 14519 time to create 1 rle with old method : 0.01726365089416504 length of segment : 110 time for calcul the mask position with numpy : 0.0005507469177246094 nb_pixel_total : 18901 time to create 1 rle with old method : 0.02193927764892578 length of segment : 151 time for calcul the mask position with numpy : 0.003465890884399414 nb_pixel_total : 181674 time to create 1 rle with new method : 0.00548553466796875 length of segment : 1080 time for calcul the mask position with numpy : 0.01722884178161621 nb_pixel_total : 705945 time to create 1 rle with new method : 0.06318330764770508 length of segment : 1746 time for calcul the mask position with numpy : 0.0023889541625976562 nb_pixel_total : 56888 time to create 1 rle with old method : 0.06393241882324219 length of segment : 625 time for calcul the mask position with numpy : 0.0005772113800048828 nb_pixel_total : 18028 time to create 1 rle with old method : 0.021735429763793945 length of segment : 159 time for calcul the mask position with numpy : 0.004491567611694336 nb_pixel_total : 173123 time to create 1 rle with new method : 0.0065822601318359375 length of segment : 626 time for calcul the mask position with numpy : 0.0004322528839111328 nb_pixel_total : 9802 time to create 1 rle with old method : 0.012177705764770508 length of segment : 96 time for calcul the mask position with numpy : 0.0009691715240478516 nb_pixel_total : 22698 time to create 1 rle with old method : 0.026986122131347656 length of segment : 201 time for calcul the mask position with numpy : 0.00019931793212890625 nb_pixel_total : 2153 time to create 1 rle with old method : 0.003844738006591797 length of segment : 47 time for calcul the mask position with numpy : 0.00344085693359375 nb_pixel_total : 62489 time to create 1 rle with old method : 0.08574032783508301 length of segment : 423 time for calcul the mask position with numpy : 0.0047931671142578125 nb_pixel_total : 163687 time to create 1 rle with new method : 0.006927967071533203 length of segment : 452 time for calcul the mask position with numpy : 0.000659942626953125 nb_pixel_total : 23568 time to create 1 rle with old method : 0.0286867618560791 length of segment : 235 time for calcul the mask position with numpy : 0.00557398796081543 nb_pixel_total : 197591 time to create 1 rle with new method : 0.00871419906616211 length of segment : 748 time for calcul the mask position with numpy : 0.00020599365234375 nb_pixel_total : 1799 time to create 1 rle with old method : 0.002307415008544922 length of segment : 61 time for calcul the mask position with numpy : 0.0009388923645019531 nb_pixel_total : 21854 time to create 1 rle with old method : 0.02971649169921875 length of segment : 138 time for calcul the mask position with numpy : 0.00152587890625 nb_pixel_total : 48076 time to create 1 rle with old method : 0.05564737319946289 length of segment : 341 time for calcul the mask position with numpy : 0.00018215179443359375 nb_pixel_total : 2285 time to create 1 rle with old method : 0.002880573272705078 length of segment : 78 time for calcul the mask position with numpy : 0.02049851417541504 nb_pixel_total : 654084 time to create 1 rle with new method : 0.05114555358886719 length of segment : 1371 time for calcul the mask position with numpy : 0.0005714893341064453 nb_pixel_total : 13173 time to create 1 rle with old method : 0.015821218490600586 length of segment : 169 time for calcul the mask position with numpy : 0.0001742839813232422 nb_pixel_total : 2074 time to create 1 rle with old method : 0.0027387142181396484 length of segment : 68 time for calcul the mask position with numpy : 0.0024912357330322266 nb_pixel_total : 78212 time to create 1 rle with old method : 0.09103512763977051 length of segment : 365 time for calcul the mask position with numpy : 0.0007648468017578125 nb_pixel_total : 19133 time to create 1 rle with old method : 0.02200460433959961 length of segment : 182 time for calcul the mask position with numpy : 0.0018393993377685547 nb_pixel_total : 28660 time to create 1 rle with old method : 0.03399372100830078 length of segment : 523 time for calcul the mask position with numpy : 0.0036139488220214844 nb_pixel_total : 99504 time to create 1 rle with old method : 0.14188671112060547 length of segment : 360 time for calcul the mask position with numpy : 0.0002853870391845703 nb_pixel_total : 4401 time to create 1 rle with old method : 0.00555109977722168 length of segment : 101 time for calcul the mask position with numpy : 0.00464320182800293 nb_pixel_total : 168880 time to create 1 rle with new method : 0.006788969039916992 length of segment : 581 time for calcul the mask position with numpy : 0.04214000701904297 nb_pixel_total : 1465717 time to create 1 rle with new method : 0.42668771743774414 length of segment : 2322 time for calcul the mask position with numpy : 0.024741649627685547 nb_pixel_total : 633085 time to create 1 rle with new method : 0.1221318244934082 length of segment : 2162 time for calcul the mask position with numpy : 0.0026502609252929688 nb_pixel_total : 137089 time to create 1 rle with old method : 0.15261054039001465 length of segment : 442 time for calcul the mask position with numpy : 0.0029294490814208984 nb_pixel_total : 120828 time to create 1 rle with old method : 0.13637280464172363 length of segment : 831 time for calcul the mask position with numpy : 0.0006282329559326172 nb_pixel_total : 29159 time to create 1 rle with old method : 0.03533673286437988 length of segment : 186 time for calcul the mask position with numpy : 0.004585981369018555 nb_pixel_total : 157726 time to create 1 rle with new method : 0.008321762084960938 length of segment : 449 time for calcul the mask position with numpy : 0.0015916824340820312 nb_pixel_total : 51272 time to create 1 rle with old method : 0.059678077697753906 length of segment : 340 time for calcul the mask position with numpy : 0.006558418273925781 nb_pixel_total : 201234 time to create 1 rle with new method : 0.019117355346679688 length of segment : 719 time for calcul the mask position with numpy : 0.0038895606994628906 nb_pixel_total : 189726 time to create 1 rle with new method : 0.007186412811279297 length of segment : 470 time for calcul the mask position with numpy : 0.007287263870239258 nb_pixel_total : 452203 time to create 1 rle with new method : 0.028717756271362305 length of segment : 1191 time for calcul the mask position with numpy : 0.004394054412841797 nb_pixel_total : 207385 time to create 1 rle with new method : 0.010353803634643555 length of segment : 807 time for calcul the mask position with numpy : 0.0006229877471923828 nb_pixel_total : 8606 time to create 1 rle with old method : 0.010728836059570312 length of segment : 185 time for calcul the mask position with numpy : 0.0006580352783203125 nb_pixel_total : 14503 time to create 1 rle with old method : 0.017946958541870117 length of segment : 151 time for calcul the mask position with numpy : 0.001447916030883789 nb_pixel_total : 29587 time to create 1 rle with old method : 0.038530826568603516 length of segment : 326 time for calcul the mask position with numpy : 0.0019414424896240234 nb_pixel_total : 58269 time to create 1 rle with old method : 0.07707905769348145 length of segment : 607 time for calcul the mask position with numpy : 0.0033969879150390625 nb_pixel_total : 153611 time to create 1 rle with new method : 0.004736661911010742 length of segment : 756 time for calcul the mask position with numpy : 0.00024366378784179688 nb_pixel_total : 3587 time to create 1 rle with old method : 0.004401206970214844 length of segment : 111 time for calcul the mask position with numpy : 0.0003399848937988281 nb_pixel_total : 15759 time to create 1 rle with old method : 0.018599510192871094 length of segment : 178 time for calcul the mask position with numpy : 0.005185365676879883 nb_pixel_total : 229200 time to create 1 rle with new method : 0.006937265396118164 length of segment : 593 time for calcul the mask position with numpy : 0.0005533695220947266 nb_pixel_total : 6663 time to create 1 rle with old method : 0.009577035903930664 length of segment : 100 time for calcul the mask position with numpy : 0.00048279762268066406 nb_pixel_total : 8687 time to create 1 rle with old method : 0.010474920272827148 length of segment : 250 time for calcul the mask position with numpy : 0.002415895462036133 nb_pixel_total : 98218 time to create 1 rle with old method : 0.1128084659576416 length of segment : 503 time for calcul the mask position with numpy : 0.0017573833465576172 nb_pixel_total : 82392 time to create 1 rle with old method : 0.0983576774597168 length of segment : 360 time for calcul the mask position with numpy : 0.0004923343658447266 nb_pixel_total : 12236 time to create 1 rle with old method : 0.01531362533569336 length of segment : 155 time for calcul the mask position with numpy : 0.0007581710815429688 nb_pixel_total : 18319 time to create 1 rle with old method : 0.021642208099365234 length of segment : 349 time for calcul the mask position with numpy : 0.0038182735443115234 nb_pixel_total : 173332 time to create 1 rle with new method : 0.006728172302246094 length of segment : 523 time for calcul the mask position with numpy : 0.002805948257446289 nb_pixel_total : 120568 time to create 1 rle with old method : 0.16349554061889648 length of segment : 621 time for calcul the mask position with numpy : 0.001219034194946289 nb_pixel_total : 21869 time to create 1 rle with old method : 0.03360342979431152 length of segment : 386 time for calcul the mask position with numpy : 0.0035469532012939453 nb_pixel_total : 111408 time to create 1 rle with old method : 0.15162301063537598 length of segment : 715 time for calcul the mask position with numpy : 0.0009522438049316406 nb_pixel_total : 50390 time to create 1 rle with old method : 0.058595895767211914 length of segment : 209 time for calcul the mask position with numpy : 0.003133058547973633 nb_pixel_total : 153571 time to create 1 rle with new method : 0.00529170036315918 length of segment : 597 time for calcul the mask position with numpy : 0.002560853958129883 nb_pixel_total : 89499 time to create 1 rle with old method : 0.10768389701843262 length of segment : 458 time for calcul the mask position with numpy : 0.0035865306854248047 nb_pixel_total : 146653 time to create 1 rle with old method : 0.17264962196350098 length of segment : 501 time for calcul the mask position with numpy : 0.0006194114685058594 nb_pixel_total : 19177 time to create 1 rle with old method : 0.02356243133544922 length of segment : 126 time for calcul the mask position with numpy : 0.0005848407745361328 nb_pixel_total : 13450 time to create 1 rle with old method : 0.01634693145751953 length of segment : 167 time for calcul the mask position with numpy : 0.00460362434387207 nb_pixel_total : 162130 time to create 1 rle with new method : 0.005693912506103516 length of segment : 563 time for calcul the mask position with numpy : 0.002389192581176758 nb_pixel_total : 48889 time to create 1 rle with old method : 0.06015777587890625 length of segment : 321 time for calcul the mask position with numpy : 0.0005016326904296875 nb_pixel_total : 5305 time to create 1 rle with old method : 0.0066585540771484375 length of segment : 154 time for calcul the mask position with numpy : 0.007586956024169922 nb_pixel_total : 231088 time to create 1 rle with new method : 0.013149023056030273 length of segment : 662 time for calcul the mask position with numpy : 0.0031402111053466797 nb_pixel_total : 85479 time to create 1 rle with old method : 0.1061704158782959 length of segment : 435 time for calcul the mask position with numpy : 0.003724813461303711 nb_pixel_total : 229176 time to create 1 rle with new method : 0.011775732040405273 length of segment : 664 time for calcul the mask position with numpy : 0.0021598339080810547 nb_pixel_total : 78605 time to create 1 rle with old method : 0.09076070785522461 length of segment : 461 time for calcul the mask position with numpy : 0.0018177032470703125 nb_pixel_total : 31869 time to create 1 rle with old method : 0.03787589073181152 length of segment : 349 time for calcul the mask position with numpy : 0.00023293495178222656 nb_pixel_total : 5810 time to create 1 rle with old method : 0.0071239471435546875 length of segment : 155 time for calcul the mask position with numpy : 0.0012485980987548828 nb_pixel_total : 54370 time to create 1 rle with old method : 0.06784224510192871 length of segment : 284 time for calcul the mask position with numpy : 0.0012745857238769531 nb_pixel_total : 82262 time to create 1 rle with old method : 0.09655261039733887 length of segment : 446 time for calcul the mask position with numpy : 0.004288911819458008 nb_pixel_total : 172351 time to create 1 rle with new method : 0.006732940673828125 length of segment : 597 time for calcul the mask position with numpy : 0.0008997917175292969 nb_pixel_total : 23879 time to create 1 rle with old method : 0.02875065803527832 length of segment : 182 time for calcul the mask position with numpy : 0.0009288787841796875 nb_pixel_total : 23818 time to create 1 rle with old method : 0.02937150001525879 length of segment : 208 time for calcul the mask position with numpy : 0.013680696487426758 nb_pixel_total : 662318 time to create 1 rle with new method : 0.028680801391601562 length of segment : 1503 time for calcul the mask position with numpy : 0.0001819133758544922 nb_pixel_total : 5416 time to create 1 rle with old method : 0.006639719009399414 length of segment : 121 time for calcul the mask position with numpy : 0.0019178390502929688 nb_pixel_total : 78790 time to create 1 rle with old method : 0.09532499313354492 length of segment : 312 time for calcul the mask position with numpy : 0.012238502502441406 nb_pixel_total : 368298 time to create 1 rle with new method : 0.023932456970214844 length of segment : 1582 time for calcul the mask position with numpy : 0.0003008842468261719 nb_pixel_total : 2648 time to create 1 rle with old method : 0.0034570693969726562 length of segment : 90 time for calcul the mask position with numpy : 0.017786741256713867 nb_pixel_total : 738860 time to create 1 rle with new method : 0.15230512619018555 length of segment : 2327 time for calcul the mask position with numpy : 0.001918792724609375 nb_pixel_total : 51607 time to create 1 rle with old method : 0.0623774528503418 length of segment : 339 time for calcul the mask position with numpy : 0.014875173568725586 nb_pixel_total : 444962 time to create 1 rle with new method : 0.034735679626464844 length of segment : 825 time for calcul the mask position with numpy : 0.0005612373352050781 nb_pixel_total : 12327 time to create 1 rle with old method : 0.017979860305786133 length of segment : 124 time for calcul the mask position with numpy : 0.0005831718444824219 nb_pixel_total : 19902 time to create 1 rle with old method : 0.024526596069335938 length of segment : 130 time for calcul the mask position with numpy : 0.002244710922241211 nb_pixel_total : 68595 time to create 1 rle with old method : 0.08232808113098145 length of segment : 235 time for calcul the mask position with numpy : 0.001096487045288086 nb_pixel_total : 52798 time to create 1 rle with old method : 0.07888960838317871 length of segment : 346 time spent for convertir_results : 23.854833126068115 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 144 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 75363 save missing photos in datou_result : time spend for datou_step_exec : 82.24078416824341 time spend to save output : 115.77098655700684 total time spend for step 1 : 198.01177072525024 step2:crop_condition Sat Mar 8 14:55:49 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 144 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 17 About to insert : list_path_to_insert length 17 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 12 About to insert : list_path_to_insert length 12 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 34 About to insert : list_path_to_insert length 34 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1338867245, 1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690] Looping around the photos to save general results len do output : 84 /-3699508972Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3699508978Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663078Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663095Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663112Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663108Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663104Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663106Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663125Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663119Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663123Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663139Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663142Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663173Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663185Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663191Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663187Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3699508975Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3699508977Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663089Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663085Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663090Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663162Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663172Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663179Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663178Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663176Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663194Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663190Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663088Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3699797810Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663148Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663147Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663145Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3699508981Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3699508979Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3699508980Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663086Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3699797807Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663080Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3699797806Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663087Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3699797809Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663097Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663100Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663111Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663105Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663110Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663117Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663124Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663126Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663137Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663136Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663121Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663128Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663143Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663161Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663158Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663169Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663170Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663168Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663180Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663184Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663182Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3699797812Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663198Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663197Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663186Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3699508974Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663103Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663107Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663120Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663129Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663159Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663157Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663174Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3699508976Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663093Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663081Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663144Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663175Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663181Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663199Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663135Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1338867245', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335238113', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237999', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237967', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237931', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237877', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237738', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237734', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237716', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237690', None, None, None, None, None, '2652113') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 262 time used for this insertion : 0.04737544059753418 save_final save missing photos in datou_result : time spend for datou_step_exec : 70.4122154712677 time spend to save output : 0.17052125930786133 total time spend for step 2 : 70.58273673057556 step3:thcl Sat Mar 8 14:57:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.015190362930297852 time to convert the images to numpy array : 0.40461277961730957 time to import caffe and check if the image exist : 0.020312070846557617 time to convert the images to numpy array : 0.4432382583618164 time to import caffe and check if the image exist : 0.02145862579345703 time to convert the images to numpy array : 0.8521876335144043 time to import caffe and check if the image exist : 0.014847517013549805 time to convert the images to numpy array : 0.8794338703155518 time to import caffe and check if the image exist : 0.0418093204498291 time to convert the images to numpy array : 0.8546037673950195 time to import caffe and check if the image exist : 0.015563726425170898 time to convert the images to numpy array : 0.8884541988372803 time to import caffe and check if the image exist : 0.0224916934967041 time to convert the images to numpy array : 1.1680400371551514 time to import caffe and check if the image exist : 0.012769699096679688 time to convert the images to numpy array : 1.1838054656982422 time to import caffe and check if the image exist : 0.01836228370666504 time to convert the images to numpy array : 1.1840808391571045 time to import caffe and check if the image exist : 0.015737295150756836 time to convert the images to numpy array : 1.4300379753112793 total time to convert the images to numpy array : 1.653921127319336 list photo_ids error: [] list photo_ids correct : [-3671663181, -3671663199, -3671663135, -3671663182, -3699797812, -3671663198, -3671663197, -3671663186, -3699508974, -3671663103, -3671663107, -3671663120, -3699508977, -3671663089, -3671663085, -3671663090, -3671663162, -3671663172, -3671663179, -3671663178, -3671663176, -3671663111, -3671663105, -3671663110, -3671663117, -3671663124, -3671663126, -3671663137, -3671663136, -3671663121, -3671663194, -3671663190, -3671663088, -3699797810, -3671663148, -3671663147, -3671663145, -3699508981, -3699508979, -3671663129, -3671663159, -3671663157, -3671663174, -3699508976, -3671663093, -3671663081, -3671663144, -3671663175, -3671663128, -3671663143, -3671663161, -3671663158, -3671663169, -3671663170, -3671663168, -3671663180, -3671663184, -3699508972, -3699508978, -3671663078, -3671663095, -3671663112, -3671663108, -3671663104, -3671663106, -3671663125, -3671663119, -3671663123, -3671663139, -3671663142, -3671663173, -3671663185, -3671663191, -3671663187, -3699508975, -3699508980, -3671663086, -3699797807, -3671663080, -3699797806, -3671663087, -3699797809, -3671663097, -3671663100] number of photos to traite : 84 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 13 wait 20 seconds l 3637 free memory gpu now : 13 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.3216056823730469 time used to do the prediction : 0.3156108856201172 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5081 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5081 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.5765960216522217 time used to do the prediction : 0.16719579696655273 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.956390380859375e-05 save missing photos in datou_result : time spend for datou_step_exec : 32.1359121799469 time spend to save output : 0.0010724067687988281 total time spend for step 3 : 32.1369845867157 step4:merge_mask_thcl_custom Sat Mar 8 14:57:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 144 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1338867245, 1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690] Looping around the photos to save general results len do output : 10 /1338867245Didn't retrieve data .Didn't retrieve data . /1335238113Didn't retrieve data .Didn't retrieve data . /1335237999Didn't retrieve data .Didn't retrieve data . /1335237967Didn't retrieve data .Didn't retrieve data . /1335237931Didn't retrieve data .Didn't retrieve data . /1335237877Didn't retrieve data .Didn't retrieve data . /1335237738Didn't retrieve data .Didn't retrieve data . /1335237734Didn't retrieve data .Didn't retrieve data . /1335237716Didn't retrieve data .Didn't retrieve data . /1335237690Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1338867245', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335238113', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237999', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237967', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237931', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237877', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237738', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237734', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237716', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237690', None, None, None, None, None, '2652113') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.07648038864135742 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.322894811630249 time spend to save output : 0.07705402374267578 total time spend for step 4 : 0.3999488353729248 step5:rle_unique_nms_with_priority Sat Mar 8 14:57:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 10 nb_hashtags : 5 time to prepare the origin masks : 11.914532899856567 time for calcul the mask position with numpy : 0.5947225093841553 nb_pixel_total : 6055490 time to create 1 rle with new method : 1.287719964981079 time for calcul the mask position with numpy : 0.044577836990356445 nb_pixel_total : 23289 time to create 1 rle with old method : 0.026938915252685547 time for calcul the mask position with numpy : 0.04678225517272949 nb_pixel_total : 13717 time to create 1 rle with old method : 0.015887975692749023 time for calcul the mask position with numpy : 0.03805279731750488 nb_pixel_total : 44807 time to create 1 rle with old method : 0.05627036094665527 time for calcul the mask position with numpy : 0.02983689308166504 nb_pixel_total : 172579 time to create 1 rle with new method : 0.6947352886199951 time for calcul the mask position with numpy : 0.025564908981323242 nb_pixel_total : 21901 time to create 1 rle with old method : 0.02691793441772461 time for calcul the mask position with numpy : 0.02772688865661621 nb_pixel_total : 104703 time to create 1 rle with old method : 0.12930512428283691 time for calcul the mask position with numpy : 0.02597665786743164 nb_pixel_total : 174669 time to create 1 rle with new method : 0.7172408103942871 time for calcul the mask position with numpy : 0.32904744148254395 nb_pixel_total : 1593864 time to create 1 rle with new method : 0.6840901374816895 time for calcul the mask position with numpy : 0.02865767478942871 nb_pixel_total : 8746 time to create 1 rle with old method : 0.014103174209594727 time for calcul the mask position with numpy : 0.028743267059326172 nb_pixel_total : 80635 time to create 1 rle with old method : 0.10414934158325195 create new chi : 5.098827838897705 time to delete rle : 0.01706719398498535 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 13315 TO DO : save crop sub photo not yet done ! save time : 15.918596744537354 nb_obj : 17 nb_hashtags : 7 time to prepare the origin masks : 15.238225936889648 time for calcul the mask position with numpy : 0.8599941730499268 nb_pixel_total : 5344633 time to create 1 rle with new method : 1.0230889320373535 time for calcul the mask position with numpy : 0.05450034141540527 nb_pixel_total : 36363 time to create 1 rle with old method : 0.058080434799194336 time for calcul the mask position with numpy : 0.04176616668701172 nb_pixel_total : 24232 time to create 1 rle with old method : 0.027520179748535156 time for calcul the mask position with numpy : 0.04488229751586914 nb_pixel_total : 172896 time to create 1 rle with new method : 0.7557673454284668 time for calcul the mask position with numpy : 0.04116702079772949 nb_pixel_total : 93887 time to create 1 rle with old method : 0.10759234428405762 time for calcul the mask position with numpy : 0.04529213905334473 nb_pixel_total : 39277 time to create 1 rle with old method : 0.04536581039428711 time for calcul the mask position with numpy : 0.04397010803222656 nb_pixel_total : 16400 time to create 1 rle with old method : 0.021457910537719727 time for calcul the mask position with numpy : 0.04498457908630371 nb_pixel_total : 178702 time to create 1 rle with new method : 1.1953105926513672 time for calcul the mask position with numpy : 0.05393791198730469 nb_pixel_total : 128046 time to create 1 rle with old method : 0.17733430862426758 time for calcul the mask position with numpy : 0.04710221290588379 nb_pixel_total : 11327 time to create 1 rle with old method : 0.015314102172851562 time for calcul the mask position with numpy : 0.06495141983032227 nb_pixel_total : 1504239 time to create 1 rle with new method : 0.8413107395172119 time for calcul the mask position with numpy : 0.04597902297973633 nb_pixel_total : 189638 time to create 1 rle with new method : 0.914574146270752 time for calcul the mask position with numpy : 0.04551410675048828 nb_pixel_total : 117888 time to create 1 rle with old method : 0.14080405235290527 time for calcul the mask position with numpy : 0.043196916580200195 nb_pixel_total : 138929 time to create 1 rle with old method : 0.1710066795349121 time for calcul the mask position with numpy : 0.04852485656738281 nb_pixel_total : 124420 time to create 1 rle with old method : 0.1579599380493164 time for calcul the mask position with numpy : 0.045578718185424805 nb_pixel_total : 33094 time to create 1 rle with old method : 0.0414731502532959 time for calcul the mask position with numpy : 0.04911446571350098 nb_pixel_total : 21897 time to create 1 rle with old method : 0.02560138702392578 time for calcul the mask position with numpy : 0.04906153678894043 nb_pixel_total : 118532 time to create 1 rle with old method : 0.14588570594787598 create new chi : 7.7076640129089355 time to delete rle : 0.004256725311279297 batch 1 Loaded 25 chid ids of type : 3726 Number RLEs to save : 20693 TO DO : save crop sub photo not yet done ! save time : 22.44979429244995 nb_obj : 9 nb_hashtags : 3 time to prepare the origin masks : 11.184775590896606 time for calcul the mask position with numpy : 0.15698719024658203 nb_pixel_total : 3380351 time to create 1 rle with new method : 0.693629264831543 time for calcul the mask position with numpy : 0.05240178108215332 nb_pixel_total : 788285 time to create 1 rle with new method : 0.4271230697631836 time for calcul the mask position with numpy : 0.05002737045288086 nb_pixel_total : 177262 time to create 1 rle with new method : 0.9990458488464355 time for calcul the mask position with numpy : 0.06680178642272949 nb_pixel_total : 47 time to create 1 rle with old method : 0.00023627281188964844 time for calcul the mask position with numpy : 0.05897021293640137 nb_pixel_total : 219818 time to create 1 rle with new method : 0.7287812232971191 time for calcul the mask position with numpy : 0.040595054626464844 nb_pixel_total : 1016 time to create 1 rle with old method : 0.0016036033630371094 time for calcul the mask position with numpy : 0.05305147171020508 nb_pixel_total : 129535 time to create 1 rle with old method : 0.15517234802246094 time for calcul the mask position with numpy : 0.283297061920166 nb_pixel_total : 3497941 time to create 1 rle with new method : 0.5035197734832764 time for calcul the mask position with numpy : 0.042021751403808594 nb_pixel_total : 54798 time to create 1 rle with old method : 0.06482458114624023 time for calcul the mask position with numpy : 0.04325413703918457 nb_pixel_total : 45347 time to create 1 rle with old method : 0.05142521858215332 create new chi : 4.641093969345093 time to delete rle : 0.001499176025390625 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 15423 TO DO : save crop sub photo not yet done ! save time : 11.529971837997437 nb_obj : 14 nb_hashtags : 6 time to prepare the origin masks : 13.024919748306274 time for calcul the mask position with numpy : 0.6994037628173828 nb_pixel_total : 5927643 time to create 1 rle with new method : 0.9454047679901123 time for calcul the mask position with numpy : 0.03989815711975098 nb_pixel_total : 7265 time to create 1 rle with old method : 0.016127586364746094 time for calcul the mask position with numpy : 0.04756426811218262 nb_pixel_total : 314673 time to create 1 rle with new method : 0.872429609298706 time for calcul the mask position with numpy : 0.04512357711791992 nb_pixel_total : 94977 time to create 1 rle with old method : 0.10375022888183594 time for calcul the mask position with numpy : 0.04459857940673828 nb_pixel_total : 68254 time to create 1 rle with old method : 0.07422542572021484 time for calcul the mask position with numpy : 0.04275369644165039 nb_pixel_total : 18960 time to create 1 rle with old method : 0.02253246307373047 time for calcul the mask position with numpy : 0.04157710075378418 nb_pixel_total : 149940 time to create 1 rle with old method : 0.16912412643432617 time for calcul the mask position with numpy : 0.03905463218688965 nb_pixel_total : 73698 time to create 1 rle with old method : 0.07901644706726074 time for calcul the mask position with numpy : 0.03279423713684082 nb_pixel_total : 587778 time to create 1 rle with new method : 0.8699018955230713 time for calcul the mask position with numpy : 0.037136077880859375 nb_pixel_total : 15339 time to create 1 rle with old method : 0.03411078453063965 time for calcul the mask position with numpy : 0.03204798698425293 nb_pixel_total : 11730 time to create 1 rle with old method : 0.017208337783813477 time for calcul the mask position with numpy : 0.025726318359375 nb_pixel_total : 170983 time to create 1 rle with new method : 1.2413930892944336 time for calcul the mask position with numpy : 0.02432727813720703 nb_pixel_total : 36749 time to create 1 rle with old method : 0.042569637298583984 time for calcul the mask position with numpy : 0.04201483726501465 nb_pixel_total : 757046 time to create 1 rle with new method : 0.8773844242095947 time for calcul the mask position with numpy : 0.02759265899658203 nb_pixel_total : 59365 time to create 1 rle with old method : 0.06923127174377441 create new chi : 6.812857627868652 time to delete rle : 0.0015826225280761719 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 18199 TO DO : save crop sub photo not yet done ! save time : 8.356775283813477 nb_obj : 17 nb_hashtags : 7 time to prepare the origin masks : 21.422784566879272 time for calcul the mask position with numpy : 0.5923528671264648 nb_pixel_total : 6772281 time to create 1 rle with new method : 0.712327241897583 time for calcul the mask position with numpy : 0.044902801513671875 nb_pixel_total : 15699 time to create 1 rle with old method : 0.018372058868408203 time for calcul the mask position with numpy : 0.039081573486328125 nb_pixel_total : 21609 time to create 1 rle with old method : 0.024169921875 time for calcul the mask position with numpy : 0.0446927547454834 nb_pixel_total : 19067 time to create 1 rle with old method : 0.0211029052734375 time for calcul the mask position with numpy : 0.0444033145904541 nb_pixel_total : 197204 time to create 1 rle with new method : 1.1002318859100342 time for calcul the mask position with numpy : 0.03994607925415039 nb_pixel_total : 62304 time to create 1 rle with old method : 0.08285212516784668 time for calcul the mask position with numpy : 0.04436779022216797 nb_pixel_total : 99269 time to create 1 rle with old method : 0.1379256248474121 time for calcul the mask position with numpy : 0.06527018547058105 nb_pixel_total : 2064 time to create 1 rle with old method : 0.0031332969665527344 time for calcul the mask position with numpy : 0.05392289161682129 nb_pixel_total : 163354 time to create 1 rle with new method : 0.5935425758361816 time for calcul the mask position with numpy : 0.03857088088989258 nb_pixel_total : 22032 time to create 1 rle with old method : 0.025677204132080078 time for calcul the mask position with numpy : 0.04853105545043945 nb_pixel_total : 654090 time to create 1 rle with new method : 1.0448527336120605 time for calcul the mask position with numpy : 0.056031227111816406 nb_pixel_total : 2278 time to create 1 rle with old method : 0.0037794113159179688 time for calcul the mask position with numpy : 0.052257537841796875 nb_pixel_total : 1792 time to create 1 rle with old method : 0.0020864009857177734 time for calcul the mask position with numpy : 0.04324460029602051 nb_pixel_total : 48026 time to create 1 rle with old method : 0.05559802055358887 time for calcul the mask position with numpy : 0.0394287109375 nb_pixel_total : 172938 time to create 1 rle with new method : 1.0433952808380127 time for calcul the mask position with numpy : 0.053228139877319336 nb_pixel_total : 2134 time to create 1 rle with old method : 0.002841472625732422 time for calcul the mask position with numpy : 0.04341936111450195 nb_pixel_total : 9782 time to create 1 rle with old method : 0.011319875717163086 time for calcul the mask position with numpy : 0.0456085205078125 nb_pixel_total : 28477 time to create 1 rle with old method : 0.03326988220214844 create new chi : 6.483515501022339 time to delete rle : 0.0018656253814697266 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 13733 TO DO : save crop sub photo not yet done ! save time : 2.258134126663208 nb_obj : 12 nb_hashtags : 4 time to prepare the origin masks : 17.490297317504883 time for calcul the mask position with numpy : 0.6100854873657227 nb_pixel_total : 5196737 time to create 1 rle with new method : 1.631988525390625 time for calcul the mask position with numpy : 0.047446489334106445 nb_pixel_total : 157312 time to create 1 rle with new method : 0.99800705909729 time for calcul the mask position with numpy : 0.04414534568786621 nb_pixel_total : 226013 time to create 1 rle with new method : 0.7517712116241455 time for calcul the mask position with numpy : 0.05086565017700195 nb_pixel_total : 57196 time to create 1 rle with old method : 0.07864618301391602 time for calcul the mask position with numpy : 0.045382022857666016 nb_pixel_total : 2361 time to create 1 rle with old method : 0.00452733039855957 time for calcul the mask position with numpy : 0.05993294715881348 nb_pixel_total : 633739 time to create 1 rle with new method : 0.7835886478424072 time for calcul the mask position with numpy : 0.06392908096313477 nb_pixel_total : 1474988 time to create 1 rle with new method : 0.5059194564819336 time for calcul the mask position with numpy : 0.049150943756103516 nb_pixel_total : 65586 time to create 1 rle with old method : 0.0775599479675293 time for calcul the mask position with numpy : 0.04447507858276367 nb_pixel_total : 4386 time to create 1 rle with old method : 0.005087375640869141 time for calcul the mask position with numpy : 0.047179222106933594 nb_pixel_total : 136939 time to create 1 rle with old method : 0.16382765769958496 time for calcul the mask position with numpy : 0.04680061340332031 nb_pixel_total : 120573 time to create 1 rle with old method : 0.14094996452331543 time for calcul the mask position with numpy : 0.04809141159057617 nb_pixel_total : 189572 time to create 1 rle with new method : 1.7865188121795654 time for calcul the mask position with numpy : 0.0565643310546875 nb_pixel_total : 28998 time to create 1 rle with old method : 0.043550729751586914 create new chi : 8.408365726470947 time to delete rle : 0.002719402313232422 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 20311 TO DO : save crop sub photo not yet done ! save time : 2.0865721702575684 nb_obj : 13 nb_hashtags : 5 time to prepare the origin masks : 10.687879085540771 time for calcul the mask position with numpy : 0.6274900436401367 nb_pixel_total : 7586412 time to create 1 rle with new method : 1.0029263496398926 time for calcul the mask position with numpy : 0.0455784797668457 nb_pixel_total : 82158 time to create 1 rle with old method : 0.10905313491821289 time for calcul the mask position with numpy : 0.04619097709655762 nb_pixel_total : 15701 time to create 1 rle with old method : 0.019760608673095703 time for calcul the mask position with numpy : 0.045558929443359375 nb_pixel_total : 8660 time to create 1 rle with old method : 0.011284351348876953 time for calcul the mask position with numpy : 0.056241750717163086 nb_pixel_total : 98058 time to create 1 rle with old method : 0.1254420280456543 time for calcul the mask position with numpy : 0.058110952377319336 nb_pixel_total : 228817 time to create 1 rle with new method : 0.620368242263794 time for calcul the mask position with numpy : 0.047864437103271484 nb_pixel_total : 50 time to create 1 rle with old method : 0.0001857280731201172 time for calcul the mask position with numpy : 0.05178523063659668 nb_pixel_total : 153576 time to create 1 rle with new method : 1.4531991481781006 time for calcul the mask position with numpy : 0.049813270568847656 nb_pixel_total : 3739 time to create 1 rle with old method : 0.004680156707763672 time for calcul the mask position with numpy : 0.04915142059326172 nb_pixel_total : 58129 time to create 1 rle with old method : 0.06873273849487305 time for calcul the mask position with numpy : 0.05071115493774414 nb_pixel_total : 29576 time to create 1 rle with old method : 0.03424978256225586 time for calcul the mask position with numpy : 0.05703258514404297 nb_pixel_total : 14379 time to create 1 rle with old method : 0.024184226989746094 time for calcul the mask position with numpy : 0.05066490173339844 nb_pixel_total : 8561 time to create 1 rle with old method : 0.009895801544189453 time for calcul the mask position with numpy : 0.04439544677734375 nb_pixel_total : 6584 time to create 1 rle with old method : 0.007985830307006836 create new chi : 4.906985521316528 time to delete rle : 0.0014710426330566406 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 10461 TO DO : save crop sub photo not yet done ! save time : 4.093214988708496 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 6.09436559677124 time for calcul the mask position with numpy : 0.6630687713623047 nb_pixel_total : 7835940 time to create 1 rle with new method : 1.0453119277954102 time for calcul the mask position with numpy : 0.04750466346740723 nb_pixel_total : 50067 time to create 1 rle with old method : 0.06097698211669922 time for calcul the mask position with numpy : 0.03672051429748535 nb_pixel_total : 155043 time to create 1 rle with new method : 1.7673263549804688 time for calcul the mask position with numpy : 0.06688642501831055 nb_pixel_total : 21823 time to create 1 rle with old method : 0.03428030014038086 time for calcul the mask position with numpy : 0.045981407165527344 nb_pixel_total : 111048 time to create 1 rle with old method : 0.19382810592651367 time for calcul the mask position with numpy : 0.03965950012207031 nb_pixel_total : 120479 time to create 1 rle with old method : 0.1442553997039795 create new chi : 4.245587348937988 time to delete rle : 0.001100778579711914 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 7100 TO DO : save crop sub photo not yet done ! save time : 3.199125289916992 nb_obj : 15 nb_hashtags : 8 time to prepare the origin masks : 24.866111755371094 time for calcul the mask position with numpy : 1.0538935661315918 nb_pixel_total : 7388584 time to create 1 rle with new method : 1.2048962116241455 time for calcul the mask position with numpy : 0.04371905326843262 nb_pixel_total : 6336 time to create 1 rle with old method : 0.007995128631591797 time for calcul the mask position with numpy : 0.04371953010559082 nb_pixel_total : 31743 time to create 1 rle with old method : 0.03827047348022461 time for calcul the mask position with numpy : 0.044956207275390625 nb_pixel_total : 229025 time to create 1 rle with new method : 1.1294052600860596 time for calcul the mask position with numpy : 0.04290938377380371 nb_pixel_total : 48786 time to create 1 rle with old method : 0.05380082130432129 time for calcul the mask position with numpy : 0.03675341606140137 nb_pixel_total : 54331 time to create 1 rle with old method : 0.06464385986328125 time for calcul the mask position with numpy : 0.02469921112060547 nb_pixel_total : 5792 time to create 1 rle with old method : 0.006355762481689453 time for calcul the mask position with numpy : 0.02655768394470215 nb_pixel_total : 162285 time to create 1 rle with new method : 0.927750825881958 time for calcul the mask position with numpy : 0.05875730514526367 nb_pixel_total : 7898 time to create 1 rle with old method : 0.009645938873291016 time for calcul the mask position with numpy : 0.051233530044555664 nb_pixel_total : 146175 time to create 1 rle with old method : 0.1884760856628418 time for calcul the mask position with numpy : 0.05138850212097168 nb_pixel_total : 19074 time to create 1 rle with old method : 0.022146940231323242 time for calcul the mask position with numpy : 0.05497431755065918 nb_pixel_total : 5298 time to create 1 rle with old method : 0.007379770278930664 time for calcul the mask position with numpy : 0.0551607608795166 nb_pixel_total : 78823 time to create 1 rle with old method : 0.09038496017456055 time for calcul the mask position with numpy : 0.0491640567779541 nb_pixel_total : 7705 time to create 1 rle with old method : 0.009141921997070312 time for calcul the mask position with numpy : 0.0551762580871582 nb_pixel_total : 13364 time to create 1 rle with old method : 0.021891117095947266 time for calcul the mask position with numpy : 0.052092552185058594 nb_pixel_total : 89181 time to create 1 rle with old method : 0.10277700424194336 create new chi : 5.748898983001709 time to delete rle : 0.0021963119506835938 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 12338 TO DO : save crop sub photo not yet done ! save time : 13.37193775177002 nb_obj : 15 nb_hashtags : 6 time to prepare the origin masks : 18.051551818847656 time for calcul the mask position with numpy : 0.49463343620300293 nb_pixel_total : 5941926 time to create 1 rle with new method : 1.396008014678955 time for calcul the mask position with numpy : 0.0334470272064209 nb_pixel_total : 68348 time to create 1 rle with old method : 0.08024048805236816 time for calcul the mask position with numpy : 0.03196406364440918 nb_pixel_total : 23807 time to create 1 rle with old method : 0.027413606643676758 time for calcul the mask position with numpy : 0.0323793888092041 nb_pixel_total : 12280 time to create 1 rle with old method : 0.015092611312866211 time for calcul the mask position with numpy : 0.03181862831115723 nb_pixel_total : 956 time to create 1 rle with old method : 0.0013840198516845703 time for calcul the mask position with numpy : 0.03795957565307617 nb_pixel_total : 444781 time to create 1 rle with new method : 0.9168200492858887 time for calcul the mask position with numpy : 0.03308510780334473 nb_pixel_total : 52062 time to create 1 rle with old method : 0.06471133232116699 time for calcul the mask position with numpy : 0.030496835708618164 nb_pixel_total : 78675 time to create 1 rle with old method : 0.1020815372467041 time for calcul the mask position with numpy : 0.03957343101501465 nb_pixel_total : 467258 time to create 1 rle with new method : 0.524770975112915 time for calcul the mask position with numpy : 0.028438806533813477 nb_pixel_total : 19816 time to create 1 rle with old method : 0.022881746292114258 time for calcul the mask position with numpy : 0.030102968215942383 nb_pixel_total : 2633 time to create 1 rle with old method : 0.0031251907348632812 time for calcul the mask position with numpy : 0.0353701114654541 nb_pixel_total : 123536 time to create 1 rle with old method : 0.27666544914245605 time for calcul the mask position with numpy : 0.03972578048706055 nb_pixel_total : 5382 time to create 1 rle with old method : 0.006392955780029297 time for calcul the mask position with numpy : 0.051795005798339844 nb_pixel_total : 661087 time to create 1 rle with new method : 1.3380210399627686 time for calcul the mask position with numpy : 0.06162405014038086 nb_pixel_total : 368126 time to create 1 rle with new method : 0.5029847621917725 time for calcul the mask position with numpy : 0.06851363182067871 nb_pixel_total : 23727 time to create 1 rle with old method : 0.027460575103759766 create new chi : 6.591166973114014 time to delete rle : 0.0022149085998535156 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 17977 TO DO : save crop sub photo not yet done ! save time : 21.205790758132935 map_output_result : {1338867245: (0.0, 'Should be the crop_list due to order', 0), 1335238113: (0.0, 'Should be the crop_list due to order', 0), 1335237999: (0.0, 'Should be the crop_list due to order', 0), 1335237967: (0.0, 'Should be the crop_list due to order', 0), 1335237931: (0.0, 'Should be the crop_list due to order', 0), 1335237877: (0.0, 'Should be the crop_list due to order', 0), 1335237738: (0.0, 'Should be the crop_list due to order', 0), 1335237734: (0.0, 'Should be the crop_list due to order', 0), 1335237716: (0.0, 'Should be the crop_list due to order', 0), 1335237690: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1338867245, 1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690] Looping around the photos to save general results len do output : 10 /1338867245.Didn't retrieve data . /1335238113.Didn't retrieve data . /1335237999.Didn't retrieve data . /1335237967.Didn't retrieve data . /1335237931.Didn't retrieve data . /1335237877.Didn't retrieve data . /1335237738.Didn't retrieve data . /1335237734.Didn't retrieve data . /1335237716.Didn't retrieve data . /1335237690.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1338867245', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335238113', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237999', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237967', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237931', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237877', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237738', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237734', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237716', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237690', None, None, None, None, None, '2652113') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014466285705566406 save_final save missing photos in datou_result : time spend for datou_step_exec : 319.65116572380066 time spend to save output : 0.02389240264892578 total time spend for step 5 : 319.6750581264496 step6:crop_condition Sat Mar 8 15:02:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 152 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 4869462 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741442585_860455 we have uploaded 9 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.9780099391937256 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741442594_860455 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 3.347656488418579 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741442602_860455 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.868391513824463 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741442606_860455 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.0110766887664795 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741442611_860455 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 3.388651132583618 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741442618_860455 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.13938307762146 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1338867245, 1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690] Looping around the photos to save general results len do output : 33 /1341894265Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894266Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894268Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894270Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894271Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894273Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894275Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894276Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894278Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894337Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894339Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894341Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894344Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894345Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894346Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894349Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894409Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894411Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894412Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894414Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894416Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894441Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894442Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894499Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894501Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894503Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894504Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894506Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894508Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894510Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894552Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894553Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341894555Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1338867245', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335238113', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237999', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237967', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237931', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237877', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237738', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237734', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237716', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237690', None, None, None, None, None, '2652113') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 109 time used for this insertion : 0.03990006446838379 save_final save missing photos in datou_result : time spend for datou_step_exec : 58.72972273826599 time spend to save output : 0.05812811851501465 total time spend for step 6 : 58.787850856781006 step7:ventilate_hashtags_in_portfolio Sat Mar 8 15:03:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 21178362 get user id for portfolio 21178362 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21178362 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('plastique','Teint_Dans_La_Masse','mal_croppe','metal','Carton_gris','environnement','kraft','flou','cartonnette','papier','autre_refus','Carton_brun')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21178362 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('plastique','Teint_Dans_La_Masse','mal_croppe','metal','Carton_gris','environnement','kraft','flou','cartonnette','papier','autre_refus','Carton_brun')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21178362 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('plastique','Teint_Dans_La_Masse','mal_croppe','metal','Carton_gris','environnement','kraft','flou','cartonnette','papier','autre_refus','Carton_brun')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/21182680,21182681,21182682,21182683,21182684,21182685,21182686,21182687,21182688,21182689,21182690,21182691?tags=plastique,Teint_Dans_La_Masse,mal_croppe,metal,Carton_gris,environnement,kraft,flou,cartonnette,papier,autre_refus,Carton_brun Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1338867245, 1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690] Looping around the photos to save general results len do output : 1 /21178362. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1338867245', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335238113', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237999', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237967', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237931', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237877', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237738', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237734', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237716', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237690', None, None, None, None, None, '2652113') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01822948455810547 save_final save missing photos in datou_result : time spend for datou_step_exec : 5.836806774139404 time spend to save output : 0.018494844436645508 total time spend for step 7 : 5.85530161857605 step8:final Sat Mar 8 15:03:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1338867245: ('0.09203158876802205',), 1335238113: ('0.09203158876802205',), 1335237999: ('0.09203158876802205',), 1335237967: ('0.09203158876802205',), 1335237931: ('0.09203158876802205',), 1335237877: ('0.09203158876802205',), 1335237738: ('0.09203158876802205',), 1335237734: ('0.09203158876802205',), 1335237716: ('0.09203158876802205',), 1335237690: ('0.09203158876802205',)} new output for save of step final : {1338867245: ('0.09203158876802205',), 1335238113: ('0.09203158876802205',), 1335237999: ('0.09203158876802205',), 1335237967: ('0.09203158876802205',), 1335237931: ('0.09203158876802205',), 1335237877: ('0.09203158876802205',), 1335237738: ('0.09203158876802205',), 1335237734: ('0.09203158876802205',), 1335237716: ('0.09203158876802205',), 1335237690: ('0.09203158876802205',)} [1338867245, 1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690] Looping around the photos to save general results len do output : 10 /1338867245.Didn't retrieve data . /1335238113.Didn't retrieve data . /1335237999.Didn't retrieve data . /1335237967.Didn't retrieve data . /1335237931.Didn't retrieve data . /1335237877.Didn't retrieve data . /1335237738.Didn't retrieve data . /1335237734.Didn't retrieve data . /1335237716.Didn't retrieve data . /1335237690.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1338867245', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335238113', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237999', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237967', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237931', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237877', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237738', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237734', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237716', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237690', None, None, None, None, None, '2652113') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01387476921081543 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.26633429527282715 time spend to save output : 0.014459848403930664 total time spend for step 8 : 0.2807941436767578 step9:velours_tree Sat Mar 8 15:03:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.2097914218902588 time spend to save output : 6.222724914550781e-05 total time spend for step 9 : 0.2098536491394043 step10:send_mail_cod Sat Mar 8 15:03:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21178362_08-03-2025_15_03_57.pdf 21182680 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211826801741442637 21182682 imagette211826821741442638 21182683 imagette211826831741442638 21182686 change filename to text .imagette211826861741442638 21182687 imagette211826871741442638 21182688 change filename to text .change filename to text .change filename to text .imagette211826881741442638 21182690 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211826901741442639 21182681 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211826811741442640 21182684 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211826841741442641 21182691 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211826911741442642 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21178362 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21182680,21182681,21182682,21182683,21182684,21182685,21182686,21182687,21182688,21182689,21182690,21182691?tags=plastique,Teint_Dans_La_Masse,mal_croppe,metal,Carton_gris,environnement,kraft,flou,cartonnette,papier,autre_refus,Carton_brun your option no_mail is active, we will not send the real mail to your client args[1338867245] : ('0.09203158876802205',) no score found for photo 1338867245 We are sending mail with results at report@fotonower.com args[1335238113] : ('0.09203158876802205',) no score found for photo 1335238113 We are sending mail with results at report@fotonower.com args[1335237999] : ('0.09203158876802205',) no score found for photo 1335237999 We are sending mail with results at report@fotonower.com args[1335237967] : ('0.09203158876802205',) no score found for photo 1335237967 We are sending mail with results at report@fotonower.com args[1335237931] : ('0.09203158876802205',) no score found for photo 1335237931 We are sending mail with results at report@fotonower.com args[1335237877] : ('0.09203158876802205',) no score found for photo 1335237877 We are sending mail with results at report@fotonower.com args[1335237738] : ('0.09203158876802205',) no score found for photo 1335237738 We are sending mail with results at report@fotonower.com args[1335237734] : ('0.09203158876802205',) no score found for photo 1335237734 We are sending mail with results at report@fotonower.com args[1335237716] : ('0.09203158876802205',) no score found for photo 1335237716 We are sending mail with results at report@fotonower.com args[1335237690] : ('0.09203158876802205',) no score found for photo 1335237690 We are sending mail with results at report@fotonower.com refus_total : 0.09203158876802205 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=21178362 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1335237474,1335237716,1335237734,1335237738,1335237931,1335237967,1335237612,1335237669,1335237465,1335237522,1335237690,1335237877,1335237999,1335238113,1338867245) Found this number of photos: 15 begin to download photo : 1335237474 begin to download photo : 1335237931 begin to download photo : 1335237465 begin to download photo : 1335237999 download finish for photo 1335237931 begin to download photo : 1335237967 download finish for photo 1335237474 begin to download photo : 1335237716 download finish for photo 1335237999 begin to download photo : 1335238113 download finish for photo 1335237465 begin to download photo : 1335237522 download finish for photo 1335237716 begin to download photo : 1335237734 download finish for photo 1335238113 begin to download photo : 1338867245 download finish for photo 1335237967 begin to download photo : 1335237612 download finish for photo 1335237522 begin to download photo : 1335237690 download finish for photo 1335237734 begin to download photo : 1335237738 download finish for photo 1335237612 begin to download photo : 1335237669 download finish for photo 1338867245 download finish for photo 1335237690 begin to download photo : 1335237877 download finish for photo 1335237738 download finish for photo 1335237669 download finish for photo 1335237877 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21178362_08-03-2025_15_03_57.pdf results_Auto_P21178362_08-03-2025_15_03_57.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21178362_08-03-2025_15_03_57.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','21178362','results_Auto_P21178362_08-03-2025_15_03_57.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21178362_08-03-2025_15_03_57.pdf','pdf','','1.13','0.09203158876802205') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1338867245, 1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1338867245', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335238113', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237999', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237967', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237931', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237877', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237738', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237734', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237716', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237690', None, None, None, None, None, '2652113') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.017912626266479492 save_final save missing photos in datou_result : time spend for datou_step_exec : 10.299482822418213 time spend to save output : 0.01813650131225586 total time spend for step 10 : 10.317619323730469 step11:split_time_score Sat Mar 8 15:04:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('17', 15),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 06022025 21178362 Nombre de photos uploadées : 15 / 23040 (0%) 06022025 21178362 Nombre de photos taguées (types de déchets): 0 / 15 (0%) 06022025 21178362 Nombre de photos taguées (volume) : 0 / 15 (0%) elapsed_time : load_data_split_time_score 2.1457672119140625e-06 elapsed_time : order_list_meta_photo_and_scores 7.62939453125e-06 ??????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0009145736694335938 elapsed_time : insert_dashboard_record_day_entry 0.05511164665222168 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.012745412019500559 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21178346_08-03-2025_12_13_51.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21178346 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21178346 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21178357 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21178358 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21178359 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21178361 order by id desc limit 1 Qualite : 0.09203158876802205 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21178362_08-03-2025_15_03_57.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21178362 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21178362 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21178364 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21178365 order by id desc limit 1 Qualite : 0.057806851809158274 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21178366_08-03-2025_12_06_51.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21178366 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21178366 AND mptpi.`type`=4207 To do Qualite : 0.05048999944828504 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21178368_08-03-2025_12_06_08.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21178368 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21178368 AND mptpi.`type`=4207 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'06022025': {'nb_upload': 15, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1338867245, 1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690] Looping around the photos to save general results len do output : 1 /21178362Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1338867245', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335238113', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237999', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237967', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237931', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237877', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237738', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237734', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237716', None, None, None, None, None, '2652113') ('3459', None, None, None, None, None, None, None, '2652113') ('3459', '21178362', '1335237690', None, None, None, None, None, '2652113') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.36873531341552734 save_final save missing photos in datou_result : time spend for datou_step_exec : 5.508594989776611 time spend to save output : 0.36893320083618164 total time spend for step 11 : 5.877528190612793 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 189.70user 220.53system 11:50.35elapsed 57%CPU (0avgtext+0avgdata 4209876maxresident)k 2767944inputs+165896outputs (33105major+18959830minor)pagefaults 0swaps