python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2324686 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 39>10 please execute split_portfolio.py -i 20783013 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2610904'] with mtr_portfolio_ids : ['20783013'] and first list_photo_ids : [] new path : /proc/2324686/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.6198337078094482 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Mon Feb 24 22:12:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10593 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-24 22:12:31.661235: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-24 22:12:31.687047: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-24 22:12:31.689308: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8f38000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-24 22:12:31.689369: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-24 22:12:31.693316: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-24 22:12:31.914827: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3b025660 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-24 22:12:31.914873: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-24 22:12:31.916224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-24 22:12:31.916599: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-24 22:12:31.919642: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-24 22:12:31.922541: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-24 22:12:31.923050: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-24 22:12:31.925631: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-24 22:12:31.926669: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-24 22:12:31.931064: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-24 22:12:31.932581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-24 22:12:31.932658: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-24 22:12:31.933453: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-24 22:12:31.933468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-24 22:12:31.933477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-24 22:12:31.934764: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9815 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-24 22:12:32.190949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-24 22:12:32.191099: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-24 22:12:32.191129: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-24 22:12:32.191155: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-24 22:12:32.191179: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-24 22:12:32.191204: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-24 22:12:32.191227: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-24 22:12:32.191252: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-24 22:12:32.192646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-24 22:12:32.193704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-24 22:12:32.193733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-24 22:12:32.193748: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-24 22:12:32.193762: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-24 22:12:32.193776: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-24 22:12:32.193790: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-24 22:12:32.193803: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-24 22:12:32.193817: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-24 22:12:32.194981: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-24 22:12:32.195014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-24 22:12:32.195023: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-24 22:12:32.195030: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-24 22:12:32.196206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9815 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-24 22:12:42.444684: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-24 22:12:42.617066: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 17 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.05755 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 6 Detection mask done ! Trying to reset tf kernel 2325195 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5304 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10593 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 1.033545970916748 nb_pixel_total : 4919436 time to create 1 rle with new method : 0.9734382629394531 length of segment : 4296 time for calcul the mask position with numpy : 0.0033698081970214844 nb_pixel_total : 168354 time to create 1 rle with new method : 0.0066034793853759766 length of segment : 512 time for calcul the mask position with numpy : 0.00015020370483398438 nb_pixel_total : 2071 time to create 1 rle with old method : 0.0025124549865722656 length of segment : 49 time for calcul the mask position with numpy : 0.0013628005981445312 nb_pixel_total : 37740 time to create 1 rle with old method : 0.04224681854248047 length of segment : 284 time for calcul the mask position with numpy : 0.004756450653076172 nb_pixel_total : 198199 time to create 1 rle with new method : 0.010418891906738281 length of segment : 550 time for calcul the mask position with numpy : 0.0005857944488525391 nb_pixel_total : 11714 time to create 1 rle with old method : 0.018676042556762695 length of segment : 200 time for calcul the mask position with numpy : 0.026137828826904297 nb_pixel_total : 879435 time to create 1 rle with new method : 0.180466890335083 length of segment : 2296 time for calcul the mask position with numpy : 0.003710031509399414 nb_pixel_total : 119025 time to create 1 rle with old method : 0.12683391571044922 length of segment : 610 time for calcul the mask position with numpy : 0.00399017333984375 nb_pixel_total : 191992 time to create 1 rle with new method : 0.006354808807373047 length of segment : 418 time for calcul the mask position with numpy : 0.01468038558959961 nb_pixel_total : 650533 time to create 1 rle with new method : 0.052490234375 length of segment : 1120 time for calcul the mask position with numpy : 0.0025153160095214844 nb_pixel_total : 189357 time to create 1 rle with new method : 0.00527644157409668 length of segment : 423 time for calcul the mask position with numpy : 0.002307891845703125 nb_pixel_total : 158415 time to create 1 rle with new method : 0.005514860153198242 length of segment : 770 time for calcul the mask position with numpy : 0.0002753734588623047 nb_pixel_total : 5359 time to create 1 rle with old method : 0.0065839290618896484 length of segment : 102 time for calcul the mask position with numpy : 0.0002529621124267578 nb_pixel_total : 3359 time to create 1 rle with old method : 0.004067182540893555 length of segment : 116 time for calcul the mask position with numpy : 0.0002503395080566406 nb_pixel_total : 10895 time to create 1 rle with old method : 0.013211965560913086 length of segment : 124 time for calcul the mask position with numpy : 0.0003418922424316406 nb_pixel_total : 6291 time to create 1 rle with old method : 0.007482767105102539 length of segment : 125 time for calcul the mask position with numpy : 0.0007922649383544922 nb_pixel_total : 28180 time to create 1 rle with old method : 0.03309512138366699 length of segment : 250 time for calcul the mask position with numpy : 0.0046885013580322266 nb_pixel_total : 236349 time to create 1 rle with new method : 0.012488842010498047 length of segment : 807 time for calcul the mask position with numpy : 0.0005898475646972656 nb_pixel_total : 15686 time to create 1 rle with old method : 0.018033504486083984 length of segment : 152 time for calcul the mask position with numpy : 0.003153085708618164 nb_pixel_total : 262890 time to create 1 rle with new method : 0.0076601505279541016 length of segment : 779 time for calcul the mask position with numpy : 0.0015654563903808594 nb_pixel_total : 64758 time to create 1 rle with old method : 0.07193207740783691 length of segment : 316 time for calcul the mask position with numpy : 0.0394742488861084 nb_pixel_total : 1388324 time to create 1 rle with new method : 0.1495656967163086 length of segment : 1643 time for calcul the mask position with numpy : 0.002136707305908203 nb_pixel_total : 88775 time to create 1 rle with old method : 0.09986305236816406 length of segment : 431 time for calcul the mask position with numpy : 0.013583660125732422 nb_pixel_total : 632387 time to create 1 rle with new method : 0.051198720932006836 length of segment : 1891 time for calcul the mask position with numpy : 0.0045757293701171875 nb_pixel_total : 198761 time to create 1 rle with new method : 0.00765538215637207 length of segment : 742 time for calcul the mask position with numpy : 0.0006678104400634766 nb_pixel_total : 31123 time to create 1 rle with old method : 0.036295175552368164 length of segment : 548 time for calcul the mask position with numpy : 0.00010752677917480469 nb_pixel_total : 3828 time to create 1 rle with old method : 0.004496335983276367 length of segment : 55 time for calcul the mask position with numpy : 0.01750349998474121 nb_pixel_total : 747527 time to create 1 rle with new method : 0.22839689254760742 length of segment : 1271 time for calcul the mask position with numpy : 0.0005633831024169922 nb_pixel_total : 22633 time to create 1 rle with old method : 0.030637741088867188 length of segment : 269 time for calcul the mask position with numpy : 8.296966552734375e-05 nb_pixel_total : 1583 time to create 1 rle with old method : 0.002010822296142578 length of segment : 44 time for calcul the mask position with numpy : 0.0001933574676513672 nb_pixel_total : 2245 time to create 1 rle with old method : 0.002820730209350586 length of segment : 64 time for calcul the mask position with numpy : 0.00038695335388183594 nb_pixel_total : 7558 time to create 1 rle with old method : 0.008899688720703125 length of segment : 110 time for calcul the mask position with numpy : 0.0013234615325927734 nb_pixel_total : 30527 time to create 1 rle with old method : 0.03940844535827637 length of segment : 293 time for calcul the mask position with numpy : 0.0004067420959472656 nb_pixel_total : 9413 time to create 1 rle with old method : 0.012279272079467773 length of segment : 126 time for calcul the mask position with numpy : 0.00020647048950195312 nb_pixel_total : 2739 time to create 1 rle with old method : 0.0032744407653808594 length of segment : 98 time for calcul the mask position with numpy : 0.002390623092651367 nb_pixel_total : 45922 time to create 1 rle with old method : 0.05673789978027344 length of segment : 749 time for calcul the mask position with numpy : 0.001430511474609375 nb_pixel_total : 28941 time to create 1 rle with old method : 0.03493332862854004 length of segment : 217 time for calcul the mask position with numpy : 0.002808094024658203 nb_pixel_total : 133967 time to create 1 rle with old method : 0.14667034149169922 length of segment : 350 time for calcul the mask position with numpy : 0.003073453903198242 nb_pixel_total : 133367 time to create 1 rle with old method : 0.1485309600830078 length of segment : 419 time for calcul the mask position with numpy : 0.003993988037109375 nb_pixel_total : 186001 time to create 1 rle with new method : 0.009370803833007812 length of segment : 457 time for calcul the mask position with numpy : 0.0002791881561279297 nb_pixel_total : 6168 time to create 1 rle with old method : 0.008120059967041016 length of segment : 72 time for calcul the mask position with numpy : 0.0005440711975097656 nb_pixel_total : 3628 time to create 1 rle with old method : 0.004693746566772461 length of segment : 173 time for calcul the mask position with numpy : 0.0008156299591064453 nb_pixel_total : 32764 time to create 1 rle with old method : 0.03921866416931152 length of segment : 235 time for calcul the mask position with numpy : 0.00010466575622558594 nb_pixel_total : 761 time to create 1 rle with old method : 0.0009143352508544922 length of segment : 53 time for calcul the mask position with numpy : 0.0005249977111816406 nb_pixel_total : 24764 time to create 1 rle with old method : 0.0404512882232666 length of segment : 153 time for calcul the mask position with numpy : 0.00027942657470703125 nb_pixel_total : 5107 time to create 1 rle with old method : 0.008710384368896484 length of segment : 81 time for calcul the mask position with numpy : 0.00021409988403320312 nb_pixel_total : 4458 time to create 1 rle with old method : 0.005179405212402344 length of segment : 79 time for calcul the mask position with numpy : 0.00021409988403320312 nb_pixel_total : 5574 time to create 1 rle with old method : 0.0077457427978515625 length of segment : 95 time for calcul the mask position with numpy : 0.00016236305236816406 nb_pixel_total : 2543 time to create 1 rle with old method : 0.0030975341796875 length of segment : 70 time for calcul the mask position with numpy : 0.00020885467529296875 nb_pixel_total : 6355 time to create 1 rle with old method : 0.0072994232177734375 length of segment : 79 time for calcul the mask position with numpy : 0.0004115104675292969 nb_pixel_total : 7591 time to create 1 rle with old method : 0.00919651985168457 length of segment : 118 time for calcul the mask position with numpy : 0.0006372928619384766 nb_pixel_total : 18004 time to create 1 rle with old method : 0.01963067054748535 length of segment : 220 time for calcul the mask position with numpy : 0.00022935867309570312 nb_pixel_total : 1434 time to create 1 rle with old method : 0.0017230510711669922 length of segment : 97 time for calcul the mask position with numpy : 0.0006933212280273438 nb_pixel_total : 21701 time to create 1 rle with old method : 0.024158716201782227 length of segment : 217 time for calcul the mask position with numpy : 0.0008819103240966797 nb_pixel_total : 23407 time to create 1 rle with old method : 0.026753664016723633 length of segment : 243 time for calcul the mask position with numpy : 0.004022121429443359 nb_pixel_total : 193438 time to create 1 rle with new method : 0.006007671356201172 length of segment : 744 time for calcul the mask position with numpy : 0.00029778480529785156 nb_pixel_total : 9218 time to create 1 rle with old method : 0.011461019515991211 length of segment : 114 time for calcul the mask position with numpy : 0.0003039836883544922 nb_pixel_total : 8498 time to create 1 rle with old method : 0.010366439819335938 length of segment : 106 time for calcul the mask position with numpy : 0.0006256103515625 nb_pixel_total : 18317 time to create 1 rle with old method : 0.024607181549072266 length of segment : 239 time for calcul the mask position with numpy : 0.00023317337036132812 nb_pixel_total : 1224 time to create 1 rle with old method : 0.0024428367614746094 length of segment : 42 time for calcul the mask position with numpy : 0.004164934158325195 nb_pixel_total : 137268 time to create 1 rle with old method : 0.15394234657287598 length of segment : 401 time for calcul the mask position with numpy : 0.00020313262939453125 nb_pixel_total : 1328 time to create 1 rle with old method : 0.0016651153564453125 length of segment : 65 time for calcul the mask position with numpy : 0.0006456375122070312 nb_pixel_total : 16330 time to create 1 rle with old method : 0.02215409278869629 length of segment : 191 time for calcul the mask position with numpy : 0.00021529197692871094 nb_pixel_total : 6427 time to create 1 rle with old method : 0.010524749755859375 length of segment : 111 time for calcul the mask position with numpy : 0.0006535053253173828 nb_pixel_total : 14165 time to create 1 rle with old method : 0.022578001022338867 length of segment : 114 time for calcul the mask position with numpy : 0.0004596710205078125 nb_pixel_total : 15628 time to create 1 rle with old method : 0.017910242080688477 length of segment : 200 time for calcul the mask position with numpy : 0.0006842613220214844 nb_pixel_total : 22275 time to create 1 rle with old method : 0.02668285369873047 length of segment : 152 time for calcul the mask position with numpy : 0.0001995563507080078 nb_pixel_total : 4517 time to create 1 rle with old method : 0.00570988655090332 length of segment : 54 time for calcul the mask position with numpy : 0.0002181529998779297 nb_pixel_total : 3465 time to create 1 rle with old method : 0.004217624664306641 length of segment : 103 time for calcul the mask position with numpy : 0.0043871402740478516 nb_pixel_total : 147981 time to create 1 rle with old method : 0.16336846351623535 length of segment : 491 time for calcul the mask position with numpy : 0.0005474090576171875 nb_pixel_total : 12935 time to create 1 rle with old method : 0.014323234558105469 length of segment : 152 time for calcul the mask position with numpy : 0.002373933792114258 nb_pixel_total : 75365 time to create 1 rle with old method : 0.08395934104919434 length of segment : 314 time for calcul the mask position with numpy : 0.0009980201721191406 nb_pixel_total : 12325 time to create 1 rle with old method : 0.014026641845703125 length of segment : 177 time for calcul the mask position with numpy : 0.0005252361297607422 nb_pixel_total : 8952 time to create 1 rle with old method : 0.011019468307495117 length of segment : 279 time for calcul the mask position with numpy : 0.00029659271240234375 nb_pixel_total : 8199 time to create 1 rle with old method : 0.009740352630615234 length of segment : 179 time for calcul the mask position with numpy : 0.006078958511352539 nb_pixel_total : 197992 time to create 1 rle with new method : 0.009337902069091797 length of segment : 703 time for calcul the mask position with numpy : 0.0020401477813720703 nb_pixel_total : 26640 time to create 1 rle with old method : 0.032294273376464844 length of segment : 594 time for calcul the mask position with numpy : 0.0007686614990234375 nb_pixel_total : 25538 time to create 1 rle with old method : 0.0283966064453125 length of segment : 131 time for calcul the mask position with numpy : 0.0030651092529296875 nb_pixel_total : 84561 time to create 1 rle with old method : 0.0958707332611084 length of segment : 431 time for calcul the mask position with numpy : 0.004984617233276367 nb_pixel_total : 173986 time to create 1 rle with new method : 0.0077228546142578125 length of segment : 384 time for calcul the mask position with numpy : 0.0005693435668945312 nb_pixel_total : 27097 time to create 1 rle with old method : 0.03751730918884277 length of segment : 143 time for calcul the mask position with numpy : 0.002107858657836914 nb_pixel_total : 71254 time to create 1 rle with old method : 0.07638072967529297 length of segment : 500 time for calcul the mask position with numpy : 0.0019965171813964844 nb_pixel_total : 52376 time to create 1 rle with old method : 0.0580289363861084 length of segment : 583 time for calcul the mask position with numpy : 0.0003952980041503906 nb_pixel_total : 13698 time to create 1 rle with old method : 0.01595306396484375 length of segment : 135 time for calcul the mask position with numpy : 0.0002536773681640625 nb_pixel_total : 8838 time to create 1 rle with old method : 0.009518623352050781 length of segment : 196 time for calcul the mask position with numpy : 0.0030274391174316406 nb_pixel_total : 198706 time to create 1 rle with new method : 0.0068171024322509766 length of segment : 435 time for calcul the mask position with numpy : 0.00201416015625 nb_pixel_total : 22287 time to create 1 rle with old method : 0.02547001838684082 length of segment : 413 time for calcul the mask position with numpy : 0.00038433074951171875 nb_pixel_total : 11351 time to create 1 rle with old method : 0.013019084930419922 length of segment : 209 time for calcul the mask position with numpy : 0.0003337860107421875 nb_pixel_total : 8213 time to create 1 rle with old method : 0.008829593658447266 length of segment : 190 time for calcul the mask position with numpy : 0.001089334487915039 nb_pixel_total : 51391 time to create 1 rle with old method : 0.05590367317199707 length of segment : 217 time for calcul the mask position with numpy : 0.006247043609619141 nb_pixel_total : 414767 time to create 1 rle with new method : 0.01164102554321289 length of segment : 841 time for calcul the mask position with numpy : 0.2407393455505371 nb_pixel_total : 2584065 time to create 1 rle with new method : 0.25022459030151367 length of segment : 2429 time for calcul the mask position with numpy : 0.0001609325408935547 nb_pixel_total : 2057 time to create 1 rle with old method : 0.0027751922607421875 length of segment : 37 time for calcul the mask position with numpy : 8.630752563476562e-05 nb_pixel_total : 1951 time to create 1 rle with old method : 0.002354860305786133 length of segment : 61 time for calcul the mask position with numpy : 0.26963257789611816 nb_pixel_total : 4121516 time to create 1 rle with new method : 0.5141482353210449 length of segment : 3458 time for calcul the mask position with numpy : 0.00012803077697753906 nb_pixel_total : 3269 time to create 1 rle with old method : 0.004256725311279297 length of segment : 69 time for calcul the mask position with numpy : 0.0006604194641113281 nb_pixel_total : 36762 time to create 1 rle with old method : 0.04925799369812012 length of segment : 336 time for calcul the mask position with numpy : 0.0005598068237304688 nb_pixel_total : 26071 time to create 1 rle with old method : 0.0349583625793457 length of segment : 129 time for calcul the mask position with numpy : 0.0004696846008300781 nb_pixel_total : 22137 time to create 1 rle with old method : 0.026503562927246094 length of segment : 413 time for calcul the mask position with numpy : 0.004759788513183594 nb_pixel_total : 354703 time to create 1 rle with new method : 0.010000467300415039 length of segment : 616 time for calcul the mask position with numpy : 0.0022394657135009766 nb_pixel_total : 188957 time to create 1 rle with new method : 0.00426030158996582 length of segment : 558 time for calcul the mask position with numpy : 0.15225458145141602 nb_pixel_total : 3729294 time to create 1 rle with new method : 0.7150311470031738 length of segment : 3952 time for calcul the mask position with numpy : 0.0005586147308349609 nb_pixel_total : 29488 time to create 1 rle with old method : 0.03484296798706055 length of segment : 238 time for calcul the mask position with numpy : 0.00014257431030273438 nb_pixel_total : 6806 time to create 1 rle with old method : 0.008422374725341797 length of segment : 114 time for calcul the mask position with numpy : 0.0019185543060302734 nb_pixel_total : 143325 time to create 1 rle with old method : 0.16338610649108887 length of segment : 520 time for calcul the mask position with numpy : 0.0005097389221191406 nb_pixel_total : 25901 time to create 1 rle with old method : 0.03112053871154785 length of segment : 185 time spent for convertir_results : 19.583958387374878 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 106 chid ids of type : 3663 Number RLEs to save : 49509 save missing photos in datou_result : time spend for datou_step_exec : 66.90259265899658 time spend to save output : 3.0625112056732178 total time spend for step 1 : 69.9651038646698 step2:crop_condition Mon Feb 24 22:13:38 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 106 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 27 About to insert : list_path_to_insert length 27 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1339411074, 1339409768, 1339409764, 1339409756, 1339409670, 1339409665, 1339409662, 1339409641, 1339409639, 1339409635] Looping around the photos to save general results len do output : 59 /-3685658577Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658605Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658649Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658647Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658675Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658582Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658588Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658589Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658604Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658611Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658608Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658616Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658622Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658625Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658632Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658626Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658576Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658600Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658637Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658659Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658650Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658674Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658573Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658580Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658579Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658590Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658583Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658596Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658597Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658609Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658607Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658623Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658630Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658634Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658636Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658627Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658658Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658648Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658651Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658655Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658653Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658656Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658643Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658652Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658646Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658641Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658661Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658668Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658676Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658603Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658621Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658619Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658635Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658587Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658591Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658614Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658610Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658638Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3685658671Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339411074', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409768', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409764', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409756', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409670', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409665', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409662', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409641', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409639', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409635', None, None, None, None, None, '2610904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 187 time used for this insertion : 0.028743505477905273 save_final save missing photos in datou_result : time spend for datou_step_exec : 29.949021816253662 time spend to save output : 0.03133273124694824 total time spend for step 2 : 29.98035454750061 step3:thcl Mon Feb 24 22:14:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.012577533721923828 time to convert the images to numpy array : 0.08192181587219238 time to import caffe and check if the image exist : 0.009372949600219727 time to convert the images to numpy array : 0.09646010398864746 time to import caffe and check if the image exist : 0.010935306549072266 time to convert the images to numpy array : 0.09880876541137695 time to import caffe and check if the image exist : 0.007268667221069336 time to convert the images to numpy array : 0.10401678085327148 time to import caffe and check if the image exist : 0.0053822994232177734 time to convert the images to numpy array : 0.11826419830322266 time to import caffe and check if the image exist : 0.0074880123138427734 time to convert the images to numpy array : 0.13675308227539062 time to import caffe and check if the image exist : 0.006330966949462891 time to convert the images to numpy array : 0.4489402770996094 time to import caffe and check if the image exist : 0.009808063507080078 time to convert the images to numpy array : 0.4476356506347656 time to import caffe and check if the image exist : 0.006463527679443359 time to convert the images to numpy array : 0.4610311985015869 time to import caffe and check if the image exist : 0.008956432342529297 time to convert the images to numpy array : 0.4611213207244873 total time to convert the images to numpy array : 0.6975739002227783 list photo_ids error: [] list photo_ids correct : [-3685658676, -3685658603, -3685658621, -3685658619, -3685658635, -3685658587, -3685658643, -3685658652, -3685658646, -3685658641, -3685658661, -3685658668, -3685658607, -3685658623, -3685658630, -3685658634, -3685658636, -3685658627, -3685658658, -3685658648, -3685658651, -3685658655, -3685658653, -3685658656, -3685658577, -3685658605, -3685658649, -3685658647, -3685658675, -3685658582, -3685658588, -3685658589, -3685658604, -3685658611, -3685658608, -3685658616, -3685658637, -3685658659, -3685658650, -3685658674, -3685658573, -3685658580, -3685658622, -3685658625, -3685658632, -3685658626, -3685658576, -3685658600, -3685658579, -3685658590, -3685658583, -3685658596, -3685658597, -3685658609, -3685658591, -3685658614, -3685658610, -3685658638, -3685658671] number of photos to traite : 59 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10593 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10374 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.6250777244567871 time used to do the prediction : 0.15694475173950195 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10372 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10372 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.6777362823486328 time used to do the prediction : 0.11346268653869629 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.0503997802734375e-05 save missing photos in datou_result : time spend for datou_step_exec : 9.450208187103271 time spend to save output : 0.00044155120849609375 total time spend for step 3 : 9.450649738311768 step4:merge_mask_thcl_custom Mon Feb 24 22:14:18 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 106 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1339411074, 1339409768, 1339409764, 1339409756, 1339409670, 1339409665, 1339409662, 1339409641, 1339409639, 1339409635] Looping around the photos to save general results len do output : 10 /1339411074Didn't retrieve data .Didn't retrieve data . /1339409768Didn't retrieve data .Didn't retrieve data . /1339409764Didn't retrieve data .Didn't retrieve data . /1339409756Didn't retrieve data .Didn't retrieve data . /1339409670Didn't retrieve data .Didn't retrieve data . /1339409665Didn't retrieve data .Didn't retrieve data . /1339409662Didn't retrieve data .Didn't retrieve data . /1339409641Didn't retrieve data .Didn't retrieve data . /1339409639Didn't retrieve data .Didn't retrieve data . /1339409635Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339411074', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409768', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409764', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409756', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409670', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409665', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409662', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409641', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409639', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409635', None, None, None, None, None, '2610904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013606071472167969 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.0577852725982666 time spend to save output : 0.01411890983581543 total time spend for step 4 : 0.07190418243408203 step5:rle_unique_nms_with_priority Mon Feb 24 22:14:18 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 2 nb_hashtags : 2 time to prepare the origin masks : 2.2535862922668457 time for calcul the mask position with numpy : 0.034108877182006836 nb_pixel_total : 1980479 time to create 1 rle with new method : 0.6921975612640381 time for calcul the mask position with numpy : 0.34394121170043945 nb_pixel_total : 4901737 time to create 1 rle with new method : 0.7365517616271973 time for calcul the mask position with numpy : 0.023407459259033203 nb_pixel_total : 168024 time to create 1 rle with new method : 0.5767173767089844 create new chi : 2.481559991836548 time to delete rle : 0.018021106719970703 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 11442 TO DO : save crop sub photo not yet done ! save time : 0.6103982925415039 nb_obj : 17 nb_hashtags : 6 time to prepare the origin masks : 10.48920202255249 time for calcul the mask position with numpy : 0.5040602684020996 nb_pixel_total : 4639801 time to create 1 rle with new method : 1.994572639465332 time for calcul the mask position with numpy : 0.03542733192443848 nb_pixel_total : 17216 time to create 1 rle with old method : 0.018159151077270508 time for calcul the mask position with numpy : 0.03385329246520996 nb_pixel_total : 6272 time to create 1 rle with old method : 0.006703853607177734 time for calcul the mask position with numpy : 0.03479456901550293 nb_pixel_total : 10753 time to create 1 rle with old method : 0.011867761611938477 time for calcul the mask position with numpy : 0.03714394569396973 nb_pixel_total : 197285 time to create 1 rle with new method : 1.4267144203186035 time for calcul the mask position with numpy : 0.0331881046295166 nb_pixel_total : 3340 time to create 1 rle with old method : 0.0036115646362304688 time for calcul the mask position with numpy : 0.03302574157714844 nb_pixel_total : 15620 time to create 1 rle with old method : 0.016876935958862305 time for calcul the mask position with numpy : 0.030567407608032227 nb_pixel_total : 17050 time to create 1 rle with old method : 0.017482519149780273 time for calcul the mask position with numpy : 0.029907703399658203 nb_pixel_total : 118512 time to create 1 rle with old method : 0.12301754951477051 time for calcul the mask position with numpy : 0.03314709663391113 nb_pixel_total : 5831 time to create 1 rle with old method : 0.007006645202636719 time for calcul the mask position with numpy : 0.03934884071350098 nb_pixel_total : 650486 time to create 1 rle with new method : 0.7637417316436768 time for calcul the mask position with numpy : 0.035481929779052734 nb_pixel_total : 11683 time to create 1 rle with old method : 0.012895345687866211 time for calcul the mask position with numpy : 0.024229049682617188 nb_pixel_total : 97030 time to create 1 rle with old method : 0.11204981803894043 time for calcul the mask position with numpy : 0.03832530975341797 nb_pixel_total : 188975 time to create 1 rle with new method : 1.6126513481140137 time for calcul the mask position with numpy : 0.04342985153198242 nb_pixel_total : 872770 time to create 1 rle with new method : 0.9471004009246826 time for calcul the mask position with numpy : 0.034250497817993164 nb_pixel_total : 37620 time to create 1 rle with old method : 0.041788578033447266 time for calcul the mask position with numpy : 0.0352330207824707 nb_pixel_total : 157945 time to create 1 rle with new method : 0.7202613353729248 time for calcul the mask position with numpy : 0.0342254638671875 nb_pixel_total : 2051 time to create 1 rle with old method : 0.002347707748413086 create new chi : 9.082512855529785 time to delete rle : 0.0016427040100097656 batch 1 Loaded 18 chid ids of type : 3726 Number RLEs to save : 18136 TO DO : save crop sub photo not yet done ! save time : 1.2723288536071777 nb_obj : 10 nb_hashtags : 4 time to prepare the origin masks : 9.168146848678589 time for calcul the mask position with numpy : 0.4947049617767334 nb_pixel_total : 4464448 time to create 1 rle with new method : 1.3470985889434814 time for calcul the mask position with numpy : 0.03522229194641113 nb_pixel_total : 1561 time to create 1 rle with old method : 0.0017609596252441406 time for calcul the mask position with numpy : 0.0345308780670166 nb_pixel_total : 22636 time to create 1 rle with old method : 0.023789644241333008 time for calcul the mask position with numpy : 0.03597450256347656 nb_pixel_total : 156889 time to create 1 rle with new method : 0.6816003322601318 time for calcul the mask position with numpy : 0.035460472106933594 nb_pixel_total : 3818 time to create 1 rle with old method : 0.004523038864135742 time for calcul the mask position with numpy : 0.044277191162109375 nb_pixel_total : 631949 time to create 1 rle with new method : 0.8852031230926514 time for calcul the mask position with numpy : 0.04216599464416504 nb_pixel_total : 198497 time to create 1 rle with new method : 0.7243790626525879 time for calcul the mask position with numpy : 0.03362011909484863 nb_pixel_total : 88532 time to create 1 rle with old method : 0.09777283668518066 time for calcul the mask position with numpy : 0.04896283149719238 nb_pixel_total : 1386216 time to create 1 rle with new method : 0.5883207321166992 time for calcul the mask position with numpy : 0.03362417221069336 nb_pixel_total : 64631 time to create 1 rle with old method : 0.06666302680969238 time for calcul the mask position with numpy : 0.03420257568359375 nb_pixel_total : 31063 time to create 1 rle with old method : 0.03286933898925781 create new chi : 5.457611799240112 time to delete rle : 0.0018639564514160156 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 15116 TO DO : save crop sub photo not yet done ! save time : 2.0270140171051025 nb_obj : 15 nb_hashtags : 6 time to prepare the origin masks : 10.496686458587646 time for calcul the mask position with numpy : 0.5977137088775635 nb_pixel_total : 6533073 time to create 1 rle with new method : 0.5810766220092773 time for calcul the mask position with numpy : 0.038781166076660156 nb_pixel_total : 30496 time to create 1 rle with old method : 0.03316307067871094 time for calcul the mask position with numpy : 0.036454200744628906 nb_pixel_total : 762 time to create 1 rle with old method : 0.0010418891906738281 time for calcul the mask position with numpy : 0.035358428955078125 nb_pixel_total : 3575 time to create 1 rle with old method : 0.0039217472076416016 time for calcul the mask position with numpy : 0.026621580123901367 nb_pixel_total : 28817 time to create 1 rle with old method : 0.03278398513793945 time for calcul the mask position with numpy : 0.02299356460571289 nb_pixel_total : 133090 time to create 1 rle with old method : 0.1752634048461914 time for calcul the mask position with numpy : 0.021364212036132812 nb_pixel_total : 7530 time to create 1 rle with old method : 0.008493661880493164 time for calcul the mask position with numpy : 0.021338462829589844 nb_pixel_total : 6146 time to create 1 rle with old method : 0.007241964340209961 time for calcul the mask position with numpy : 0.022152185440063477 nb_pixel_total : 24647 time to create 1 rle with old method : 0.05440545082092285 time for calcul the mask position with numpy : 0.02211308479309082 nb_pixel_total : 32746 time to create 1 rle with old method : 0.03772449493408203 time for calcul the mask position with numpy : 0.022461414337158203 nb_pixel_total : 4464 time to create 1 rle with old method : 0.005389213562011719 time for calcul the mask position with numpy : 0.022727251052856445 nb_pixel_total : 9355 time to create 1 rle with old method : 0.010159015655517578 time for calcul the mask position with numpy : 0.03540205955505371 nb_pixel_total : 2728 time to create 1 rle with old method : 0.0028944015502929688 time for calcul the mask position with numpy : 0.034105777740478516 nb_pixel_total : 45734 time to create 1 rle with old method : 0.048955440521240234 time for calcul the mask position with numpy : 0.03710532188415527 nb_pixel_total : 184845 time to create 1 rle with new method : 0.7595789432525635 time for calcul the mask position with numpy : 0.034863948822021484 nb_pixel_total : 2232 time to create 1 rle with old method : 0.0025250911712646484 create new chi : 2.8649816513061523 time to delete rle : 0.0009322166442871094 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 8817 TO DO : save crop sub photo not yet done ! save time : 1.0392699241638184 nb_obj : 7 nb_hashtags : 5 time to prepare the origin masks : 5.523392200469971 time for calcul the mask position with numpy : 0.7472007274627686 nb_pixel_total : 7000837 time to create 1 rle with new method : 1.195260763168335 time for calcul the mask position with numpy : 0.022613525390625 nb_pixel_total : 7563 time to create 1 rle with old method : 0.008336544036865234 time for calcul the mask position with numpy : 0.02291703224182129 nb_pixel_total : 5548 time to create 1 rle with old method : 0.006113767623901367 time for calcul the mask position with numpy : 0.02422189712524414 nb_pixel_total : 17936 time to create 1 rle with old method : 0.01920294761657715 time for calcul the mask position with numpy : 0.021472930908203125 nb_pixel_total : 2529 time to create 1 rle with old method : 0.002901792526245117 time for calcul the mask position with numpy : 0.022367000579833984 nb_pixel_total : 4438 time to create 1 rle with old method : 0.005464792251586914 time for calcul the mask position with numpy : 0.021420955657958984 nb_pixel_total : 6331 time to create 1 rle with old method : 0.007143259048461914 time for calcul the mask position with numpy : 0.02190232276916504 nb_pixel_total : 5058 time to create 1 rle with old method : 0.0055773258209228516 create new chi : 2.1952054500579834 time to delete rle : 0.0004737377166748047 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 3666 TO DO : save crop sub photo not yet done ! save time : 0.24934172630310059 nb_obj : 16 nb_hashtags : 5 time to prepare the origin masks : 11.534574270248413 time for calcul the mask position with numpy : 0.27074551582336426 nb_pixel_total : 6571122 time to create 1 rle with new method : 0.8328561782836914 time for calcul the mask position with numpy : 0.02215409278869629 nb_pixel_total : 124527 time to create 1 rle with old method : 0.1341538429260254 time for calcul the mask position with numpy : 0.022513866424560547 nb_pixel_total : 16328 time to create 1 rle with old method : 0.02121567726135254 time for calcul the mask position with numpy : 0.021088123321533203 nb_pixel_total : 4516 time to create 1 rle with old method : 0.0052356719970703125 time for calcul the mask position with numpy : 0.021692991256713867 nb_pixel_total : 13990 time to create 1 rle with old method : 0.01595759391784668 time for calcul the mask position with numpy : 0.022986173629760742 nb_pixel_total : 14139 time to create 1 rle with old method : 0.015193939208984375 time for calcul the mask position with numpy : 0.02124762535095215 nb_pixel_total : 18299 time to create 1 rle with old method : 0.019465208053588867 time for calcul the mask position with numpy : 0.020341873168945312 nb_pixel_total : 1320 time to create 1 rle with old method : 0.0015263557434082031 time for calcul the mask position with numpy : 0.02349233627319336 nb_pixel_total : 192212 time to create 1 rle with new method : 0.9270012378692627 time for calcul the mask position with numpy : 0.025996923446655273 nb_pixel_total : 1216 time to create 1 rle with old method : 0.0014109611511230469 time for calcul the mask position with numpy : 0.022125244140625 nb_pixel_total : 8435 time to create 1 rle with old method : 0.011388778686523438 time for calcul the mask position with numpy : 0.03948092460632324 nb_pixel_total : 9117 time to create 1 rle with old method : 0.010539770126342773 time for calcul the mask position with numpy : 0.03444194793701172 nb_pixel_total : 23325 time to create 1 rle with old method : 0.02534198760986328 time for calcul the mask position with numpy : 0.035544633865356445 nb_pixel_total : 21664 time to create 1 rle with old method : 0.0336301326751709 time for calcul the mask position with numpy : 0.03870034217834473 nb_pixel_total : 6399 time to create 1 rle with old method : 0.010625600814819336 time for calcul the mask position with numpy : 0.03424835205078125 nb_pixel_total : 1431 time to create 1 rle with old method : 0.001588582992553711 time for calcul the mask position with numpy : 0.028389692306518555 nb_pixel_total : 22200 time to create 1 rle with old method : 0.023989200592041016 create new chi : 2.8583099842071533 time to delete rle : 0.0008802413940429688 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 8134 TO DO : save crop sub photo not yet done ! save time : 0.6737120151519775 nb_obj : 20 nb_hashtags : 5 time to prepare the origin masks : 15.691098928451538 time for calcul the mask position with numpy : 1.304410457611084 nb_pixel_total : 6085439 time to create 1 rle with new method : 0.5219616889953613 time for calcul the mask position with numpy : 0.020667314529418945 nb_pixel_total : 12269 time to create 1 rle with old method : 0.014183521270751953 time for calcul the mask position with numpy : 0.021785736083984375 nb_pixel_total : 66356 time to create 1 rle with old method : 0.07767009735107422 time for calcul the mask position with numpy : 0.02208423614501953 nb_pixel_total : 185 time to create 1 rle with old method : 0.0004932880401611328 time for calcul the mask position with numpy : 0.022193193435668945 nb_pixel_total : 52 time to create 1 rle with old method : 0.00016832351684570312 time for calcul the mask position with numpy : 0.020740509033203125 nb_pixel_total : 11252 time to create 1 rle with old method : 0.012279033660888672 time for calcul the mask position with numpy : 0.02070140838623047 nb_pixel_total : 26840 time to create 1 rle with old method : 0.029546022415161133 time for calcul the mask position with numpy : 0.0245058536529541 nb_pixel_total : 198418 time to create 1 rle with new method : 0.6299664974212646 time for calcul the mask position with numpy : 0.021629810333251953 nb_pixel_total : 71204 time to create 1 rle with old method : 0.08081436157226562 time for calcul the mask position with numpy : 0.021453142166137695 nb_pixel_total : 13635 time to create 1 rle with old method : 0.015051841735839844 time for calcul the mask position with numpy : 0.031693220138549805 nb_pixel_total : 40671 time to create 1 rle with old method : 0.043988943099975586 time for calcul the mask position with numpy : 0.03497576713562012 nb_pixel_total : 83924 time to create 1 rle with old method : 0.09153175354003906 time for calcul the mask position with numpy : 0.037470340728759766 nb_pixel_total : 197747 time to create 1 rle with new method : 0.8827006816864014 time for calcul the mask position with numpy : 0.03592824935913086 nb_pixel_total : 22133 time to create 1 rle with old method : 0.024497032165527344 time for calcul the mask position with numpy : 0.036722660064697266 nb_pixel_total : 8916 time to create 1 rle with old method : 0.009959936141967773 time for calcul the mask position with numpy : 0.021841764450073242 nb_pixel_total : 26624 time to create 1 rle with old method : 0.029432296752929688 time for calcul the mask position with numpy : 0.022382020950317383 nb_pixel_total : 147818 time to create 1 rle with old method : 0.16094756126403809 time for calcul the mask position with numpy : 0.025086402893066406 nb_pixel_total : 12252 time to create 1 rle with old method : 0.014794349670410156 time for calcul the mask position with numpy : 0.021336078643798828 nb_pixel_total : 12897 time to create 1 rle with old method : 0.014548778533935547 time for calcul the mask position with numpy : 0.02252054214477539 nb_pixel_total : 8145 time to create 1 rle with old method : 0.009190082550048828 time for calcul the mask position with numpy : 0.021312475204467773 nb_pixel_total : 3463 time to create 1 rle with old method : 0.003878355026245117 create new chi : 4.565966367721558 time to delete rle : 0.0013086795806884766 batch 1 Loaded 21 chid ids of type : 3726 Number RLEs to save : 13912 TO DO : save crop sub photo not yet done ! save time : 0.7528865337371826 nb_obj : 6 nb_hashtags : 2 time to prepare the origin masks : 5.7727134227752686 time for calcul the mask position with numpy : 0.2134537696838379 nb_pixel_total : 4376170 time to create 1 rle with new method : 0.5232529640197754 time for calcul the mask position with numpy : 0.02804732322692871 nb_pixel_total : 1312 time to create 1 rle with old method : 0.0014700889587402344 time for calcul the mask position with numpy : 0.02272200584411621 nb_pixel_total : 2027 time to create 1 rle with old method : 0.0024230480194091797 time for calcul the mask position with numpy : 0.3374488353729248 nb_pixel_total : 2196855 time to create 1 rle with new method : 0.5210065841674805 time for calcul the mask position with numpy : 0.02597212791442871 nb_pixel_total : 414486 time to create 1 rle with new method : 0.6324057579040527 time for calcul the mask position with numpy : 0.02180647850036621 nb_pixel_total : 8205 time to create 1 rle with old method : 0.009471654891967773 time for calcul the mask position with numpy : 0.022586345672607422 nb_pixel_total : 51185 time to create 1 rle with old method : 0.05598568916320801 create new chi : 2.5059244632720947 time to delete rle : 0.0008680820465087891 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 8993 TO DO : save crop sub photo not yet done ! save time : 0.5145659446716309 nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 4.9653496742248535 time for calcul the mask position with numpy : 0.24532055854797363 nb_pixel_total : 2712399 time to create 1 rle with new method : 0.8292465209960938 time for calcul the mask position with numpy : 0.02833700180053711 nb_pixel_total : 37134 time to create 1 rle with old method : 0.04731917381286621 time for calcul the mask position with numpy : 0.02425384521484375 nb_pixel_total : 108587 time to create 1 rle with old method : 0.12289690971374512 time for calcul the mask position with numpy : 0.022341489791870117 nb_pixel_total : 22146 time to create 1 rle with old method : 0.024292945861816406 time for calcul the mask position with numpy : 0.02307295799255371 nb_pixel_total : 25977 time to create 1 rle with old method : 0.028395652770996094 time for calcul the mask position with numpy : 0.027600526809692383 nb_pixel_total : 3248 time to create 1 rle with old method : 0.0036983489990234375 time for calcul the mask position with numpy : 0.3781886100769043 nb_pixel_total : 4104506 time to create 1 rle with new method : 0.5645575523376465 time for calcul the mask position with numpy : 0.023107051849365234 nb_pixel_total : 36243 time to create 1 rle with old method : 0.040547847747802734 create new chi : 2.504775285720825 time to delete rle : 0.0012784004211425781 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 12324 TO DO : save crop sub photo not yet done ! save time : 0.7290384769439697 nb_obj : 4 nb_hashtags : 4 time to prepare the origin masks : 6.297405242919922 time for calcul the mask position with numpy : 0.13437771797180176 nb_pixel_total : 2956685 time to create 1 rle with new method : 0.711634635925293 time for calcul the mask position with numpy : 0.02250051498413086 nb_pixel_total : 26054 time to create 1 rle with old method : 0.02802300453186035 time for calcul the mask position with numpy : 0.02869892120361328 nb_pixel_total : 142838 time to create 1 rle with old method : 0.1533966064453125 time for calcul the mask position with numpy : 0.026629924774169922 nb_pixel_total : 6781 time to create 1 rle with old method : 0.0076296329498291016 time for calcul the mask position with numpy : 1.0472869873046875 nb_pixel_total : 3917882 time to create 1 rle with new method : 0.8890810012817383 create new chi : 3.124605655670166 time to delete rle : 0.0011682510375976562 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 10256 TO DO : save crop sub photo not yet done ! save time : 0.5712704658508301 map_output_result : {1339411074: (0.0, 'Should be the crop_list due to order', 0), 1339409768: (0.0, 'Should be the crop_list due to order', 0), 1339409764: (0.0, 'Should be the crop_list due to order', 0), 1339409756: (0.0, 'Should be the crop_list due to order', 0), 1339409670: (0.0, 'Should be the crop_list due to order', 0), 1339409665: (0.0, 'Should be the crop_list due to order', 0), 1339409662: (0.0, 'Should be the crop_list due to order', 0), 1339409641: (0.0, 'Should be the crop_list due to order', 0), 1339409639: (0.0, 'Should be the crop_list due to order', 0), 1339409635: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1339411074, 1339409768, 1339409764, 1339409756, 1339409670, 1339409665, 1339409662, 1339409641, 1339409639, 1339409635] Looping around the photos to save general results len do output : 10 /1339411074.Didn't retrieve data . /1339409768.Didn't retrieve data . /1339409764.Didn't retrieve data . /1339409756.Didn't retrieve data . /1339409670.Didn't retrieve data . /1339409665.Didn't retrieve data . /1339409662.Didn't retrieve data . /1339409641.Didn't retrieve data . /1339409639.Didn't retrieve data . /1339409635.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339411074', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409768', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409764', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409756', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409670', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409665', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409662', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409641', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409639', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409635', None, None, None, None, None, '2610904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01304769515991211 save_final save missing photos in datou_result : time spend for datou_step_exec : 129.27730345726013 time spend to save output : 0.013647794723510742 total time spend for step 5 : 129.29095125198364 step6:crop_condition Mon Feb 24 22:16:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 114 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740431789_2324686 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6700601577758789 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740431793_2324686 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.8251111507415771 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740431798_2324686 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.1849071979522705 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740431802_2324686 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6306190490722656 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740431804_2324686 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.496574640274048 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740431810_2324686 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.2085821628570557 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1339411074, 1339409768, 1339409764, 1339409756, 1339409670, 1339409665, 1339409662, 1339409641, 1339409639, 1339409635] Looping around the photos to save general results len do output : 20 /1339421069Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421075Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421076Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421077Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421078Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421079Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421084Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421085Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421086Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421087Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421088Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421093Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421097Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421098Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421099Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421100Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421101Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421102Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421104Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339421105Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339411074', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409768', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409764', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409756', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409670', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409665', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409662', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409641', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409639', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409635', None, None, None, None, None, '2610904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 70 time used for this insertion : 0.029187917709350586 save_final save missing photos in datou_result : time spend for datou_step_exec : 24.288984775543213 time spend to save output : 0.029908180236816406 total time spend for step 6 : 24.31889295578003 step7:ventilate_hashtags_in_portfolio Mon Feb 24 22:16:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20783013 get user id for portfolio 20783013 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20783013 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','plastique','cartonnette','Teint_Dans_La_Masse','autre_refus','flou','mal_croppe','kraft','papier','metal','Carton_brun','Carton_gris')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20783013 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','plastique','cartonnette','Teint_Dans_La_Masse','autre_refus','flou','mal_croppe','kraft','papier','metal','Carton_brun','Carton_gris')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20783013 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','plastique','cartonnette','Teint_Dans_La_Masse','autre_refus','flou','mal_croppe','kraft','papier','metal','Carton_brun','Carton_gris')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20783056,20783057,20783058,20783059,20783060,20783061,20783062,20783063,20783064,20783065,20783066,20783067?tags=environnement,plastique,cartonnette,Teint_Dans_La_Masse,autre_refus,flou,mal_croppe,kraft,papier,metal,Carton_brun,Carton_gris Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1339411074, 1339409768, 1339409764, 1339409756, 1339409670, 1339409665, 1339409662, 1339409641, 1339409639, 1339409635] Looping around the photos to save general results len do output : 1 /20783013. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339411074', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409768', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409764', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409756', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409670', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409665', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409662', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409641', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409639', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409635', None, None, None, None, None, '2610904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014571428298950195 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.7091023921966553 time spend to save output : 0.014888525009155273 total time spend for step 7 : 0.7239909172058105 step8:final Mon Feb 24 22:16:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1339411074: ('0.052401843008330756',), 1339409768: ('0.052401843008330756',), 1339409764: ('0.052401843008330756',), 1339409756: ('0.052401843008330756',), 1339409670: ('0.052401843008330756',), 1339409665: ('0.052401843008330756',), 1339409662: ('0.052401843008330756',), 1339409641: ('0.052401843008330756',), 1339409639: ('0.052401843008330756',), 1339409635: ('0.052401843008330756',)} new output for save of step final : {1339411074: ('0.052401843008330756',), 1339409768: ('0.052401843008330756',), 1339409764: ('0.052401843008330756',), 1339409756: ('0.052401843008330756',), 1339409670: ('0.052401843008330756',), 1339409665: ('0.052401843008330756',), 1339409662: ('0.052401843008330756',), 1339409641: ('0.052401843008330756',), 1339409639: ('0.052401843008330756',), 1339409635: ('0.052401843008330756',)} [1339411074, 1339409768, 1339409764, 1339409756, 1339409670, 1339409665, 1339409662, 1339409641, 1339409639, 1339409635] Looping around the photos to save general results len do output : 10 /1339411074.Didn't retrieve data . /1339409768.Didn't retrieve data . /1339409764.Didn't retrieve data . /1339409756.Didn't retrieve data . /1339409670.Didn't retrieve data . /1339409665.Didn't retrieve data . /1339409662.Didn't retrieve data . /1339409641.Didn't retrieve data . /1339409639.Didn't retrieve data . /1339409635.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339411074', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409768', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409764', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409756', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409670', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409665', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409662', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409641', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409639', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409635', None, None, None, None, None, '2610904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013721942901611328 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10900354385375977 time spend to save output : 0.016426563262939453 total time spend for step 8 : 0.12543010711669922 step9:velours_tree Mon Feb 24 22:16:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.0594022274017334 time spend to save output : 5.14984130859375e-05 total time spend for step 9 : 0.059453725814819336 step10:send_mail_cod Mon Feb 24 22:16:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20783013_24-02-2025_22_16_52.pdf 20783057 change filename to text .change filename to text .imagette207830571740431812 20783058 change filename to text .imagette207830581740431813 20783060 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette207830601740431813 20783061 imagette207830611740431813 20783062 imagette207830621740431813 20783063 imagette207830631740431813 20783065 imagette207830651740431813 20783059 change filename to text .imagette207830591740431813 20783066 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette207830661740431813 20783067 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette207830671740431813 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20783013 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20783056,20783057,20783058,20783059,20783060,20783061,20783062,20783063,20783064,20783065,20783066,20783067?tags=environnement,plastique,cartonnette,Teint_Dans_La_Masse,autre_refus,flou,mal_croppe,kraft,papier,metal,Carton_brun,Carton_gris your option no_mail is active, we will not send the real mail to your client args[1339411074] : ('0.052401843008330756',) no score found for photo 1339411074 We are sending mail with results at report@fotonower.com args[1339409768] : ('0.052401843008330756',) no score found for photo 1339409768 We are sending mail with results at report@fotonower.com args[1339409764] : ('0.052401843008330756',) no score found for photo 1339409764 We are sending mail with results at report@fotonower.com args[1339409756] : ('0.052401843008330756',) no score found for photo 1339409756 We are sending mail with results at report@fotonower.com args[1339409670] : ('0.052401843008330756',) no score found for photo 1339409670 We are sending mail with results at report@fotonower.com args[1339409665] : ('0.052401843008330756',) no score found for photo 1339409665 We are sending mail with results at report@fotonower.com args[1339409662] : ('0.052401843008330756',) no score found for photo 1339409662 We are sending mail with results at report@fotonower.com args[1339409641] : ('0.052401843008330756',) no score found for photo 1339409641 We are sending mail with results at report@fotonower.com args[1339409639] : ('0.052401843008330756',) no score found for photo 1339409639 We are sending mail with results at report@fotonower.com args[1339409635] : ('0.052401843008330756',) no score found for photo 1339409635 We are sending mail with results at report@fotonower.com refus_total : 0.052401843008330756 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20783013 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1339396486,1339409768,1339409756,1339409670,1339409662,1339409620,1339409600,1339409598,1339409593,1339409570,1339409561,1339409544,1339396921,1339396540,1339396536,1339396530,1339396527,1339396525,1339396488,1339396518) Found this number of photos: 20 begin to download photo : 1339396486 begin to download photo : 1339409620 begin to download photo : 1339409561 begin to download photo : 1339396530 download finish for photo 1339409620 begin to download photo : 1339409600 download finish for photo 1339409561 begin to download photo : 1339409544 download finish for photo 1339396530 begin to download photo : 1339396527 download finish for photo 1339396486 begin to download photo : 1339409768 download finish for photo 1339409600 begin to download photo : 1339409598 download finish for photo 1339409768 begin to download photo : 1339409756 download finish for photo 1339409544 begin to download photo : 1339396921 download finish for photo 1339396527 begin to download photo : 1339396525 download finish for photo 1339409598 begin to download photo : 1339409593 download finish for photo 1339409756 begin to download photo : 1339409670 download finish for photo 1339396921 begin to download photo : 1339396540 download finish for photo 1339396525 begin to download photo : 1339396488 download finish for photo 1339409593 begin to download photo : 1339409570 download finish for photo 1339409670 begin to download photo : 1339409662 download finish for photo 1339396488 begin to download photo : 1339396518 download finish for photo 1339396540 begin to download photo : 1339396536 download finish for photo 1339409570 download finish for photo 1339409662 download finish for photo 1339396518 download finish for photo 1339396536 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20783013_24-02-2025_22_16_52.pdf results_Auto_P20783013_24-02-2025_22_16_52.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20783013_24-02-2025_22_16_52.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20783013','results_Auto_P20783013_24-02-2025_22_16_52.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20783013_24-02-2025_22_16_52.pdf','pdf','','0.69','0.052401843008330756') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1339411074, 1339409768, 1339409764, 1339409756, 1339409670, 1339409665, 1339409662, 1339409641, 1339409639, 1339409635] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339411074', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409768', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409764', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409756', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409670', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409665', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409662', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409641', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409639', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409635', None, None, None, None, None, '2610904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.15051054954528809 save_final save missing photos in datou_result : time spend for datou_step_exec : 5.2302422523498535 time spend to save output : 0.15081191062927246 total time spend for step 10 : 5.381054162979126 step11:split_time_score Mon Feb 24 22:16:58 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('20', 39),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 24022025 20783013 Nombre de photos uploadées : 39 / 23040 (0%) 24022025 20783013 Nombre de photos taguées (types de déchets): 0 / 39 (0%) 24022025 20783013 Nombre de photos taguées (volume) : 0 / 39 (0%) elapsed_time : load_data_split_time_score 2.384185791015625e-06 elapsed_time : order_list_meta_photo_and_scores 4.76837158203125e-06 ??????????????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.001621246337890625 elapsed_time : insert_dashboard_record_day_entry 0.8071422576904297 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782976 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782977 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782978 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782979 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782980 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782981 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782982 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782983 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782984 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782985 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782986 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782987 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782988 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782989 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782990 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782991 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782992 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782993 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782994 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782995 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782996 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782997 order by id desc limit 1 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20782998_24-02-2025_22_17_05.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782998 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20782999 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783000 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783001 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783002 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783003 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783004 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783005 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783006 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783007 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783008 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783009 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783010 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783011 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783012 order by id desc limit 1 Qualite : 0.052401843008330756 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20783013_24-02-2025_22_16_52.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783013 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20783013 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783014 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20783015 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'24022025': {'nb_upload': 39, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1339411074, 1339409768, 1339409764, 1339409756, 1339409670, 1339409665, 1339409662, 1339409641, 1339409639, 1339409635] Looping around the photos to save general results len do output : 1 /20783013Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339411074', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409768', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409764', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409756', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409670', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409665', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409662', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409641', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409639', None, None, None, None, None, '2610904') ('3459', None, None, None, None, None, None, None, '2610904') ('3459', '20783013', '1339409635', None, None, None, None, None, '2610904') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013917922973632812 save_final save missing photos in datou_result : time spend for datou_step_exec : 30.855559825897217 time spend to save output : 0.014213800430297852 total time spend for step 11 : 30.869773626327515 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 110.67user 149.42system 5:05.47elapsed 85%CPU (0avgtext+0avgdata 4018260maxresident)k 134144inputs+104024outputs (4920major+10930751minor)pagefaults 0swaps