python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3340985 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 11>10 please execute split_portfolio.py -i 20544938 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2585704'] with mtr_portfolio_ids : ['20544938'] and first list_photo_ids : [] new path : /proc/3340985/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.953653335571289 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 14 07:34:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10332 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-14 07:34:33.613128: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-14 07:34:33.639312: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-14 07:34:33.641430: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ff910000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-14 07:34:33.641475: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-14 07:34:33.645044: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-14 07:34:33.782107: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xa1111a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-14 07:34:33.782160: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-14 07:34:33.782966: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-14 07:34:33.783331: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-14 07:34:33.785421: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-14 07:34:33.787591: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-14 07:34:33.787954: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-14 07:34:33.794691: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-14 07:34:33.796175: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-14 07:34:33.802878: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-14 07:34:33.804438: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-14 07:34:33.804546: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-14 07:34:33.805364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-14 07:34:33.805383: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-14 07:34:33.805394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-14 07:34:33.806725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9364 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-14 07:34:34.148745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-14 07:34:34.148881: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-14 07:34:34.148904: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-14 07:34:34.148925: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-14 07:34:34.148944: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-14 07:34:34.148962: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-14 07:34:34.148981: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-14 07:34:34.148999: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-14 07:34:34.150612: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-14 07:34:34.151929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-14 07:34:34.151970: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-14 07:34:34.151989: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-14 07:34:34.152007: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-14 07:34:34.152025: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-14 07:34:34.152043: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-14 07:34:34.152060: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-14 07:34:34.152078: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-14 07:34:34.153659: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-14 07:34:34.153695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-14 07:34:34.153706: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-14 07:34:34.153716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-14 07:34:34.156356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9364 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-14 07:34:47.614279: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-14 07:34:47.816131: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 139.85000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 140.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 138.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 145.85000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 137.85000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 2.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 141.60000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 142.85000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 17 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 1.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 144.85000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 4.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 143.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 6.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 142.85000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 11 Detection mask done ! Trying to reset tf kernel 3341293 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5043 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10332 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0024204254150390625 nb_pixel_total : 55874 time to create 1 rle with old method : 0.0693519115447998 length of segment : 244 time for calcul the mask position with numpy : 0.06855177879333496 nb_pixel_total : 358055 time to create 1 rle with new method : 0.0308687686920166 length of segment : 619 time for calcul the mask position with numpy : 0.00033092498779296875 nb_pixel_total : 12415 time to create 1 rle with old method : 0.014782905578613281 length of segment : 97 time for calcul the mask position with numpy : 0.0050008296966552734 nb_pixel_total : 158869 time to create 1 rle with new method : 0.03382563591003418 length of segment : 931 time for calcul the mask position with numpy : 0.0011754035949707031 nb_pixel_total : 87065 time to create 1 rle with old method : 0.0953366756439209 length of segment : 281 time for calcul the mask position with numpy : 0.0009641647338867188 nb_pixel_total : 67012 time to create 1 rle with old method : 0.07409119606018066 length of segment : 189 time for calcul the mask position with numpy : 0.0005810260772705078 nb_pixel_total : 29280 time to create 1 rle with old method : 0.03165388107299805 length of segment : 288 time for calcul the mask position with numpy : 0.0006959438323974609 nb_pixel_total : 38548 time to create 1 rle with old method : 0.042218923568725586 length of segment : 294 time for calcul the mask position with numpy : 0.0005843639373779297 nb_pixel_total : 37340 time to create 1 rle with old method : 0.04236435890197754 length of segment : 286 time for calcul the mask position with numpy : 0.004965305328369141 nb_pixel_total : 316433 time to create 1 rle with new method : 0.017661571502685547 length of segment : 604 time for calcul the mask position with numpy : 0.001962423324584961 nb_pixel_total : 86618 time to create 1 rle with old method : 0.09802603721618652 length of segment : 768 time for calcul the mask position with numpy : 0.0016298294067382812 nb_pixel_total : 103788 time to create 1 rle with old method : 0.11336040496826172 length of segment : 529 time for calcul the mask position with numpy : 0.0016820430755615234 nb_pixel_total : 96177 time to create 1 rle with old method : 0.10590767860412598 length of segment : 412 time for calcul the mask position with numpy : 0.0004582405090332031 nb_pixel_total : 17271 time to create 1 rle with old method : 0.019885540008544922 length of segment : 327 time for calcul the mask position with numpy : 0.0019197463989257812 nb_pixel_total : 135178 time to create 1 rle with old method : 0.15328288078308105 length of segment : 493 time for calcul the mask position with numpy : 0.003537416458129883 nb_pixel_total : 215056 time to create 1 rle with new method : 0.015605449676513672 length of segment : 555 time for calcul the mask position with numpy : 0.0011401176452636719 nb_pixel_total : 44086 time to create 1 rle with old method : 0.049456119537353516 length of segment : 324 time for calcul the mask position with numpy : 0.0013453960418701172 nb_pixel_total : 91609 time to create 1 rle with old method : 0.10677552223205566 length of segment : 433 time for calcul the mask position with numpy : 0.025234699249267578 nb_pixel_total : 425406 time to create 1 rle with new method : 0.030165672302246094 length of segment : 770 time for calcul the mask position with numpy : 0.0017991065979003906 nb_pixel_total : 146902 time to create 1 rle with old method : 0.18815112113952637 length of segment : 460 time for calcul the mask position with numpy : 0.00027823448181152344 nb_pixel_total : 10250 time to create 1 rle with old method : 0.018268346786499023 length of segment : 99 time for calcul the mask position with numpy : 0.0026750564575195312 nb_pixel_total : 127306 time to create 1 rle with old method : 0.14623260498046875 length of segment : 526 time for calcul the mask position with numpy : 0.0002040863037109375 nb_pixel_total : 8458 time to create 1 rle with old method : 0.010069131851196289 length of segment : 127 time for calcul the mask position with numpy : 0.008021354675292969 nb_pixel_total : 297809 time to create 1 rle with new method : 0.026160478591918945 length of segment : 598 time for calcul the mask position with numpy : 0.002901792526245117 nb_pixel_total : 64478 time to create 1 rle with old method : 0.07115769386291504 length of segment : 492 time for calcul the mask position with numpy : 0.00693821907043457 nb_pixel_total : 222474 time to create 1 rle with new method : 0.011032581329345703 length of segment : 971 time for calcul the mask position with numpy : 0.0009024143218994141 nb_pixel_total : 23497 time to create 1 rle with old method : 0.026813507080078125 length of segment : 260 time for calcul the mask position with numpy : 0.0009307861328125 nb_pixel_total : 25698 time to create 1 rle with old method : 0.02870774269104004 length of segment : 256 time for calcul the mask position with numpy : 0.03477883338928223 nb_pixel_total : 472323 time to create 1 rle with new method : 0.03202533721923828 length of segment : 800 time for calcul the mask position with numpy : 0.006770133972167969 nb_pixel_total : 200566 time to create 1 rle with new method : 0.46003079414367676 length of segment : 1373 time for calcul the mask position with numpy : 0.002922534942626953 nb_pixel_total : 84903 time to create 1 rle with old method : 0.09245181083679199 length of segment : 471 time for calcul the mask position with numpy : 0.002748727798461914 nb_pixel_total : 68381 time to create 1 rle with old method : 0.07628965377807617 length of segment : 378 time for calcul the mask position with numpy : 0.0026831626892089844 nb_pixel_total : 142611 time to create 1 rle with old method : 0.17409157752990723 length of segment : 360 time for calcul the mask position with numpy : 0.0009286403656005859 nb_pixel_total : 28569 time to create 1 rle with old method : 0.03439474105834961 length of segment : 149 time for calcul the mask position with numpy : 0.004488229751586914 nb_pixel_total : 150572 time to create 1 rle with new method : 0.008470773696899414 length of segment : 675 time for calcul the mask position with numpy : 0.0004899501800537109 nb_pixel_total : 11735 time to create 1 rle with old method : 0.01405787467956543 length of segment : 91 time for calcul the mask position with numpy : 0.003918170928955078 nb_pixel_total : 48033 time to create 1 rle with old method : 0.05457496643066406 length of segment : 340 time for calcul the mask position with numpy : 0.013675928115844727 nb_pixel_total : 134098 time to create 1 rle with old method : 0.15297842025756836 length of segment : 1090 time for calcul the mask position with numpy : 0.01709580421447754 nb_pixel_total : 611774 time to create 1 rle with new method : 0.1802518367767334 length of segment : 1890 time for calcul the mask position with numpy : 0.008213996887207031 nb_pixel_total : 347695 time to create 1 rle with new method : 0.03910708427429199 length of segment : 712 time for calcul the mask position with numpy : 0.004725217819213867 nb_pixel_total : 169764 time to create 1 rle with new method : 0.012206315994262695 length of segment : 1034 time for calcul the mask position with numpy : 0.003587007522583008 nb_pixel_total : 174658 time to create 1 rle with new method : 0.005619525909423828 length of segment : 652 time for calcul the mask position with numpy : 0.0005183219909667969 nb_pixel_total : 15328 time to create 1 rle with old method : 0.018543481826782227 length of segment : 118 time for calcul the mask position with numpy : 0.001355886459350586 nb_pixel_total : 69799 time to create 1 rle with old method : 0.07929873466491699 length of segment : 450 time for calcul the mask position with numpy : 0.001310110092163086 nb_pixel_total : 61254 time to create 1 rle with old method : 0.06880044937133789 length of segment : 577 time for calcul the mask position with numpy : 0.004555225372314453 nb_pixel_total : 186022 time to create 1 rle with new method : 0.010170698165893555 length of segment : 657 time for calcul the mask position with numpy : 0.004575967788696289 nb_pixel_total : 201765 time to create 1 rle with new method : 0.010298013687133789 length of segment : 634 time for calcul the mask position with numpy : 0.0036916732788085938 nb_pixel_total : 198428 time to create 1 rle with new method : 0.006367921829223633 length of segment : 461 time for calcul the mask position with numpy : 0.00110626220703125 nb_pixel_total : 59828 time to create 1 rle with old method : 0.06655168533325195 length of segment : 226 time for calcul the mask position with numpy : 0.0034580230712890625 nb_pixel_total : 144968 time to create 1 rle with old method : 0.16579627990722656 length of segment : 430 time for calcul the mask position with numpy : 0.004315376281738281 nb_pixel_total : 218266 time to create 1 rle with new method : 0.008610725402832031 length of segment : 564 time for calcul the mask position with numpy : 0.0022363662719726562 nb_pixel_total : 108618 time to create 1 rle with old method : 0.12976694107055664 length of segment : 518 time for calcul the mask position with numpy : 0.0020258426666259766 nb_pixel_total : 156103 time to create 1 rle with new method : 0.0044002532958984375 length of segment : 547 time for calcul the mask position with numpy : 0.0008676052093505859 nb_pixel_total : 42063 time to create 1 rle with old method : 0.04698634147644043 length of segment : 183 time for calcul the mask position with numpy : 0.0028367042541503906 nb_pixel_total : 94976 time to create 1 rle with old method : 0.10463976860046387 length of segment : 446 time for calcul the mask position with numpy : 0.002974987030029297 nb_pixel_total : 83392 time to create 1 rle with old method : 0.0923452377319336 length of segment : 678 time for calcul the mask position with numpy : 0.007964849472045898 nb_pixel_total : 259684 time to create 1 rle with new method : 0.024967670440673828 length of segment : 796 time for calcul the mask position with numpy : 0.007199764251708984 nb_pixel_total : 243679 time to create 1 rle with new method : 0.012489557266235352 length of segment : 1022 time for calcul the mask position with numpy : 0.0007824897766113281 nb_pixel_total : 22223 time to create 1 rle with old method : 0.0328521728515625 length of segment : 291 time for calcul the mask position with numpy : 0.0026280879974365234 nb_pixel_total : 110948 time to create 1 rle with old method : 0.12328457832336426 length of segment : 406 time for calcul the mask position with numpy : 0.00029540061950683594 nb_pixel_total : 9386 time to create 1 rle with old method : 0.010749340057373047 length of segment : 164 time for calcul the mask position with numpy : 0.00597691535949707 nb_pixel_total : 276914 time to create 1 rle with new method : 0.012444019317626953 length of segment : 685 time for calcul the mask position with numpy : 0.0068569183349609375 nb_pixel_total : 157540 time to create 1 rle with new method : 0.020537614822387695 length of segment : 1345 time for calcul the mask position with numpy : 0.0013718605041503906 nb_pixel_total : 89530 time to create 1 rle with old method : 0.10167908668518066 length of segment : 246 time for calcul the mask position with numpy : 0.001203298568725586 nb_pixel_total : 82887 time to create 1 rle with old method : 0.09689068794250488 length of segment : 227 time for calcul the mask position with numpy : 0.0033659934997558594 nb_pixel_total : 85043 time to create 1 rle with old method : 0.09924912452697754 length of segment : 344 time for calcul the mask position with numpy : 0.0035834312438964844 nb_pixel_total : 168158 time to create 1 rle with new method : 0.007580757141113281 length of segment : 725 time for calcul the mask position with numpy : 0.004785299301147461 nb_pixel_total : 255855 time to create 1 rle with new method : 0.007453203201293945 length of segment : 518 time for calcul the mask position with numpy : 0.0030431747436523438 nb_pixel_total : 166247 time to create 1 rle with new method : 0.005281209945678711 length of segment : 557 time for calcul the mask position with numpy : 0.0019450187683105469 nb_pixel_total : 80542 time to create 1 rle with old method : 0.09147858619689941 length of segment : 386 time for calcul the mask position with numpy : 0.004935741424560547 nb_pixel_total : 275995 time to create 1 rle with new method : 0.008348464965820312 length of segment : 496 time for calcul the mask position with numpy : 0.0032372474670410156 nb_pixel_total : 109713 time to create 1 rle with old method : 0.12222599983215332 length of segment : 1041 time for calcul the mask position with numpy : 0.0049326419830322266 nb_pixel_total : 147240 time to create 1 rle with old method : 0.16208624839782715 length of segment : 1203 time for calcul the mask position with numpy : 0.002112150192260742 nb_pixel_total : 72054 time to create 1 rle with old method : 0.07998943328857422 length of segment : 569 time for calcul the mask position with numpy : 0.0006604194641113281 nb_pixel_total : 29542 time to create 1 rle with old method : 0.033625125885009766 length of segment : 244 time for calcul the mask position with numpy : 0.0034935474395751953 nb_pixel_total : 134769 time to create 1 rle with old method : 0.15143442153930664 length of segment : 453 time for calcul the mask position with numpy : 0.0035669803619384766 nb_pixel_total : 166104 time to create 1 rle with new method : 0.008878231048583984 length of segment : 1092 time for calcul the mask position with numpy : 0.003565549850463867 nb_pixel_total : 195823 time to create 1 rle with new method : 0.00930333137512207 length of segment : 782 time for calcul the mask position with numpy : 0.004891157150268555 nb_pixel_total : 189001 time to create 1 rle with new method : 0.009255409240722656 length of segment : 633 time for calcul the mask position with numpy : 0.002511739730834961 nb_pixel_total : 75519 time to create 1 rle with old method : 0.08449745178222656 length of segment : 511 time for calcul the mask position with numpy : 0.018660306930541992 nb_pixel_total : 793723 time to create 1 rle with new method : 0.18257713317871094 length of segment : 1903 time for calcul the mask position with numpy : 0.0011210441589355469 nb_pixel_total : 61464 time to create 1 rle with old method : 0.07101035118103027 length of segment : 260 time for calcul the mask position with numpy : 0.0023183822631835938 nb_pixel_total : 125509 time to create 1 rle with old method : 0.14116811752319336 length of segment : 280 time for calcul the mask position with numpy : 0.0005786418914794922 nb_pixel_total : 15504 time to create 1 rle with old method : 0.018291473388671875 length of segment : 117 time for calcul the mask position with numpy : 0.005843162536621094 nb_pixel_total : 251672 time to create 1 rle with new method : 0.012385845184326172 length of segment : 894 time for calcul the mask position with numpy : 0.0002269744873046875 nb_pixel_total : 12471 time to create 1 rle with old method : 0.014625310897827148 length of segment : 96 time for calcul the mask position with numpy : 0.016271352767944336 nb_pixel_total : 700320 time to create 1 rle with new method : 0.5316483974456787 length of segment : 1097 time for calcul the mask position with numpy : 0.0006930828094482422 nb_pixel_total : 26483 time to create 1 rle with old method : 0.029429197311401367 length of segment : 250 time for calcul the mask position with numpy : 0.0014641284942626953 nb_pixel_total : 50450 time to create 1 rle with old method : 0.05705618858337402 length of segment : 357 time for calcul the mask position with numpy : 0.00743556022644043 nb_pixel_total : 360862 time to create 1 rle with new method : 0.01622605323791504 length of segment : 711 time for calcul the mask position with numpy : 0.0021333694458007812 nb_pixel_total : 84229 time to create 1 rle with old method : 0.09536027908325195 length of segment : 304 time for calcul the mask position with numpy : 0.0004849433898925781 nb_pixel_total : 15397 time to create 1 rle with old method : 0.018707990646362305 length of segment : 134 time for calcul the mask position with numpy : 0.002023935317993164 nb_pixel_total : 64117 time to create 1 rle with old method : 0.07362818717956543 length of segment : 405 time for calcul the mask position with numpy : 0.008841276168823242 nb_pixel_total : 277462 time to create 1 rle with new method : 0.01873636245727539 length of segment : 948 time for calcul the mask position with numpy : 0.003458261489868164 nb_pixel_total : 173440 time to create 1 rle with new method : 0.006018877029418945 length of segment : 518 time for calcul the mask position with numpy : 0.0018279552459716797 nb_pixel_total : 53818 time to create 1 rle with old method : 0.08311700820922852 length of segment : 358 time for calcul the mask position with numpy : 0.008029699325561523 nb_pixel_total : 343689 time to create 1 rle with new method : 0.01573348045349121 length of segment : 684 time for calcul the mask position with numpy : 0.0046956539154052734 nb_pixel_total : 198969 time to create 1 rle with new method : 0.008736371994018555 length of segment : 758 time for calcul the mask position with numpy : 0.01972055435180664 nb_pixel_total : 960976 time to create 1 rle with new method : 0.16348052024841309 length of segment : 1988 time spent for convertir_results : 16.456165313720703 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 99 chid ids of type : 3663 Number RLEs to save : 55535 save missing photos in datou_result : time spend for datou_step_exec : 76.1496376991272 time spend to save output : 13.204737424850464 total time spend for step 1 : 89.35437512397766 step2:crop_condition Fri Feb 14 07:36:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 99 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 17 About to insert : list_path_to_insert length 17 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 19 About to insert : list_path_to_insert length 19 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1337553906, 1337553903, 1337553899, 1337553895, 1337553886, 1337553884, 1337553863, 1337553860, 1337553857, 1337553852] Looping around the photos to save general results len do output : 78 /-3674685026Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685037Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685043Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685060Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685049Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685059Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685076Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685074Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685085Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685080Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685083Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685084Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685095Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685107Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685105Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685104Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685118Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685036Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685030Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685029Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685040Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685044Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685065Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685070Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685099Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685103Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685034Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685048Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685047Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685062Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685058Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685120Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685032Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685031Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685041Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685051Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685050Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685069Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685067Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685077Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685073Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685079Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685088Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685087Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685089Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685082Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685093Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685094Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685110Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685108Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685114Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685027Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685028Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685038Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685045Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685056Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685052Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685055Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685057Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685096Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685109Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685116Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685035Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685053Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685102Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685101Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685111Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685112Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685123Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685122Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685115Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685119Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685121Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685033Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685072Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685071Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685075Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3674685092Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553906', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553903', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553899', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553895', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553886', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553884', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553863', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553860', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553857', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553852', None, None, None, None, None, '2585704') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 244 time used for this insertion : 0.09470653533935547 save_final save missing photos in datou_result : time spend for datou_step_exec : 65.70144581794739 time spend to save output : 0.38663268089294434 total time spend for step 2 : 66.08807849884033 step3:thcl Fri Feb 14 07:37:06 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.016326427459716797 time to convert the images to numpy array : 0.3614461421966553 time to import caffe and check if the image exist : 0.00841975212097168 time to convert the images to numpy array : 0.383044958114624 time to import caffe and check if the image exist : 0.010861873626708984 time to convert the images to numpy array : 0.39281582832336426 time to import caffe and check if the image exist : 0.006342887878417969 time to convert the images to numpy array : 0.4116859436035156 time to import caffe and check if the image exist : 0.011838197708129883 time to convert the images to numpy array : 0.41120004653930664 time to import caffe and check if the image exist : 0.012778282165527344 time to convert the images to numpy array : 0.42997264862060547 time to import caffe and check if the image exist : 0.013632535934448242 time to convert the images to numpy array : 0.44197678565979004 time to import caffe and check if the image exist : 0.01495981216430664 time to convert the images to numpy array : 0.4743766784667969 time to import caffe and check if the image exist : 0.011466026306152344 time to convert the images to numpy array : 0.5130469799041748 time to import caffe and check if the image exist : 0.011218786239624023 time to convert the images to numpy array : 0.5779316425323486 total time to convert the images to numpy array : 0.9317667484283447 list photo_ids error: [] list photo_ids correct : [-3674685121, -3674685033, -3674685072, -3674685071, -3674685075, -3674685092, -3674685099, -3674685103, -3674685034, -3674685048, -3674685047, -3674685062, -3674685058, -3674685120, -3674685085, -3674685080, -3674685083, -3674685084, -3674685095, -3674685107, -3674685105, -3674685104, -3674685026, -3674685037, -3674685043, -3674685060, -3674685049, -3674685059, -3674685076, -3674685074, -3674685032, -3674685031, -3674685041, -3674685051, -3674685050, -3674685069, -3674685067, -3674685077, -3674685110, -3674685108, -3674685114, -3674685027, -3674685028, -3674685038, -3674685045, -3674685056, -3674685052, -3674685055, -3674685057, -3674685096, -3674685109, -3674685116, -3674685035, -3674685053, -3674685073, -3674685079, -3674685088, -3674685087, -3674685089, -3674685082, -3674685093, -3674685094, -3674685118, -3674685036, -3674685030, -3674685029, -3674685040, -3674685044, -3674685065, -3674685070, -3674685102, -3674685101, -3674685111, -3674685112, -3674685123, -3674685122, -3674685115, -3674685119] number of photos to traite : 78 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5485 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5266 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.632685899734497 time used to do the prediction : 0.18992996215820312 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10553 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10553 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.6844861507415771 time used to do the prediction : 0.14576959609985352 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.5974044799804688e-05 save missing photos in datou_result : time spend for datou_step_exec : 11.530028820037842 time spend to save output : 0.0007128715515136719 total time spend for step 3 : 11.530741691589355 step4:merge_mask_thcl_custom Fri Feb 14 07:37:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 99 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1337553906, 1337553903, 1337553899, 1337553895, 1337553886, 1337553884, 1337553863, 1337553860, 1337553857, 1337553852] Looping around the photos to save general results len do output : 10 /1337553906Didn't retrieve data .Didn't retrieve data . /1337553903Didn't retrieve data .Didn't retrieve data . /1337553899Didn't retrieve data .Didn't retrieve data . /1337553895Didn't retrieve data .Didn't retrieve data . /1337553886Didn't retrieve data .Didn't retrieve data . /1337553884Didn't retrieve data .Didn't retrieve data . /1337553863Didn't retrieve data .Didn't retrieve data . /1337553860Didn't retrieve data .Didn't retrieve data . /1337553857Didn't retrieve data .Didn't retrieve data . /1337553852Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553906', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553903', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553899', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553895', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553886', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553884', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553863', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553860', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553857', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553852', None, None, None, None, None, '2585704') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013786554336547852 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.09013509750366211 time spend to save output : 0.014234781265258789 total time spend for step 4 : 0.1043698787689209 step5:rle_unique_nms_with_priority Fri Feb 14 07:37:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 11 nb_hashtags : 5 time to prepare the origin masks : 9.943434715270996 time for calcul the mask position with numpy : 1.4080126285552979 nb_pixel_total : 7085769 time to create 1 rle with new method : 1.3436217308044434 time for calcul the mask position with numpy : 0.044862985610961914 nb_pixel_total : 902 time to create 1 rle with old method : 0.0014197826385498047 time for calcul the mask position with numpy : 0.04250049591064453 nb_pixel_total : 29171 time to create 1 rle with old method : 0.03942418098449707 time for calcul the mask position with numpy : 0.04355049133300781 nb_pixel_total : 315471 time to create 1 rle with new method : 0.9320518970489502 time for calcul the mask position with numpy : 0.03929781913757324 nb_pixel_total : 38417 time to create 1 rle with old method : 0.043357133865356445 time for calcul the mask position with numpy : 0.043715715408325195 nb_pixel_total : 159308 time to create 1 rle with new method : 0.9914708137512207 time for calcul the mask position with numpy : 0.0726323127746582 nb_pixel_total : 356532 time to create 1 rle with new method : 0.8435337543487549 time for calcul the mask position with numpy : 0.037476539611816406 nb_pixel_total : 86979 time to create 1 rle with old method : 0.09554934501647949 time for calcul the mask position with numpy : 0.032921791076660156 nb_pixel_total : 12291 time to create 1 rle with old method : 0.013896465301513672 time for calcul the mask position with numpy : 0.042676448822021484 nb_pixel_total : 66842 time to create 1 rle with old method : 0.0749208927154541 time for calcul the mask position with numpy : 0.028661012649536133 nb_pixel_total : 55704 time to create 1 rle with old method : 0.06696844100952148 time for calcul the mask position with numpy : 0.026386499404907227 nb_pixel_total : 87014 time to create 1 rle with old method : 0.0973818302154541 create new chi : 6.552717447280884 time to delete rle : 0.015363693237304688 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 10861 TO DO : save crop sub photo not yet done ! save time : 0.9683051109313965 nb_obj : 4 nb_hashtags : 4 time to prepare the origin masks : 4.301669120788574 time for calcul the mask position with numpy : 0.931133508682251 nb_pixel_total : 7744819 time to create 1 rle with new method : 0.9050283432006836 time for calcul the mask position with numpy : 0.04642748832702637 nb_pixel_total : 95968 time to create 1 rle with old method : 0.10711407661437988 time for calcul the mask position with numpy : 0.028150558471679688 nb_pixel_total : 135060 time to create 1 rle with old method : 0.15506267547607422 time for calcul the mask position with numpy : 0.04922080039978027 nb_pixel_total : 214911 time to create 1 rle with new method : 0.6378049850463867 time for calcul the mask position with numpy : 0.032442331314086914 nb_pixel_total : 103642 time to create 1 rle with old method : 0.16205096244812012 create new chi : 3.1265439987182617 time to delete rle : 0.0012197494506835938 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 6136 TO DO : save crop sub photo not yet done ! save time : 0.41286396980285645 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 6.393805742263794 time for calcul the mask position with numpy : 0.5987014770507812 nb_pixel_total : 7498184 time to create 1 rle with new method : 0.6821627616882324 time for calcul the mask position with numpy : 0.042183637619018555 nb_pixel_total : 8447 time to create 1 rle with old method : 0.013493061065673828 time for calcul the mask position with numpy : 0.050647735595703125 nb_pixel_total : 424894 time to create 1 rle with new method : 0.6781256198883057 time for calcul the mask position with numpy : 0.040934085845947266 nb_pixel_total : 145572 time to create 1 rle with old method : 0.16237568855285645 time for calcul the mask position with numpy : 0.03964352607727051 nb_pixel_total : 91330 time to create 1 rle with old method : 0.09995746612548828 time for calcul the mask position with numpy : 0.04041647911071777 nb_pixel_total : 125973 time to create 1 rle with old method : 0.17248201370239258 create new chi : 2.7099475860595703 time to delete rle : 0.0010700225830078125 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6676 TO DO : save crop sub photo not yet done ! save time : 0.4298112392425537 nb_obj : 16 nb_hashtags : 6 time to prepare the origin masks : 13.700742721557617 time for calcul the mask position with numpy : 0.4728102684020996 nb_pixel_total : 5582181 time to create 1 rle with new method : 0.5862560272216797 time for calcul the mask position with numpy : 0.030471324920654297 nb_pixel_total : 10627 time to create 1 rle with old method : 0.013876914978027344 time for calcul the mask position with numpy : 0.03404951095581055 nb_pixel_total : 64387 time to create 1 rle with old method : 0.08224773406982422 time for calcul the mask position with numpy : 0.06011819839477539 nb_pixel_total : 611642 time to create 1 rle with new method : 1.0704727172851562 time for calcul the mask position with numpy : 0.04563164710998535 nb_pixel_total : 6422 time to create 1 rle with old method : 0.008486509323120117 time for calcul the mask position with numpy : 0.044207096099853516 nb_pixel_total : 62484 time to create 1 rle with old method : 0.07662439346313477 time for calcul the mask position with numpy : 0.05022072792053223 nb_pixel_total : 150357 time to create 1 rle with new method : 1.0397005081176758 time for calcul the mask position with numpy : 0.04317665100097656 nb_pixel_total : 23410 time to create 1 rle with old method : 0.02700519561767578 time for calcul the mask position with numpy : 0.03969860076904297 nb_pixel_total : 47243 time to create 1 rle with old method : 0.05552220344543457 time for calcul the mask position with numpy : 0.039453744888305664 nb_pixel_total : 330414 time to create 1 rle with new method : 0.5189859867095947 time for calcul the mask position with numpy : 0.07117009162902832 nb_pixel_total : 430343 time to create 1 rle with new method : 0.5316798686981201 time for calcul the mask position with numpy : 0.04354977607727051 nb_pixel_total : 84748 time to create 1 rle with old method : 0.09474849700927734 time for calcul the mask position with numpy : 0.04077458381652832 nb_pixel_total : 28439 time to create 1 rle with old method : 0.03350472450256348 time for calcul the mask position with numpy : 0.03941202163696289 nb_pixel_total : 142349 time to create 1 rle with old method : 0.18375158309936523 time for calcul the mask position with numpy : 0.041098594665527344 nb_pixel_total : 297276 time to create 1 rle with new method : 1.0315344333648682 time for calcul the mask position with numpy : 0.04267168045043945 nb_pixel_total : 200015 time to create 1 rle with new method : 0.6524944305419922 time for calcul the mask position with numpy : 0.043263912200927734 nb_pixel_total : 222063 time to create 1 rle with new method : 0.8916335105895996 create new chi : 8.349460363388062 time to delete rle : 0.004025459289550781 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 21630 TO DO : save crop sub photo not yet done ! save time : 1.5733311176300049 nb_obj : 7 nb_hashtags : 4 time to prepare the origin masks : 4.608389616012573 time for calcul the mask position with numpy : 0.497408390045166 nb_pixel_total : 7470919 time to create 1 rle with new method : 0.4040827751159668 time for calcul the mask position with numpy : 0.024958133697509766 nb_pixel_total : 174562 time to create 1 rle with new method : 0.45638513565063477 time for calcul the mask position with numpy : 0.02573394775390625 nb_pixel_total : 185575 time to create 1 rle with new method : 0.4453277587890625 time for calcul the mask position with numpy : 0.025754690170288086 nb_pixel_total : 7728 time to create 1 rle with old method : 0.008682966232299805 time for calcul the mask position with numpy : 0.026867389678955078 nb_pixel_total : 201240 time to create 1 rle with new method : 0.4976511001586914 time for calcul the mask position with numpy : 0.02584099769592285 nb_pixel_total : 15153 time to create 1 rle with old method : 0.018004894256591797 time for calcul the mask position with numpy : 0.026636123657226562 nb_pixel_total : 69794 time to create 1 rle with old method : 0.07693290710449219 time for calcul the mask position with numpy : 0.042963504791259766 nb_pixel_total : 169429 time to create 1 rle with new method : 0.42169833183288574 create new chi : 3.200206995010376 time to delete rle : 0.001001119613647461 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 9723 TO DO : save crop sub photo not yet done ! save time : 0.8518574237823486 nb_obj : 6 nb_hashtags : 4 time to prepare the origin masks : 4.817427158355713 time for calcul the mask position with numpy : 0.27459025382995605 nb_pixel_total : 7525025 time to create 1 rle with new method : 0.7807772159576416 time for calcul the mask position with numpy : 0.025449275970458984 nb_pixel_total : 197998 time to create 1 rle with new method : 0.37570810317993164 time for calcul the mask position with numpy : 0.0260009765625 nb_pixel_total : 42010 time to create 1 rle with old method : 0.04793834686279297 time for calcul the mask position with numpy : 0.02572464942932129 nb_pixel_total : 144241 time to create 1 rle with old method : 0.16184163093566895 time for calcul the mask position with numpy : 0.025696277618408203 nb_pixel_total : 59684 time to create 1 rle with old method : 0.06550955772399902 time for calcul the mask position with numpy : 0.02742934226989746 nb_pixel_total : 107889 time to create 1 rle with old method : 0.12125349044799805 time for calcul the mask position with numpy : 0.028628826141357422 nb_pixel_total : 217553 time to create 1 rle with new method : 0.4757966995239258 create new chi : 2.5675511360168457 time to delete rle : 0.0007915496826171875 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 6958 TO DO : save crop sub photo not yet done ! save time : 0.44963598251342773 nb_obj : 12 nb_hashtags : 4 time to prepare the origin masks : 12.643518686294556 time for calcul the mask position with numpy : 2.0855908393859863 nb_pixel_total : 6774715 time to create 1 rle with new method : 1.1892497539520264 time for calcul the mask position with numpy : 0.028612375259399414 nb_pixel_total : 9908 time to create 1 rle with old method : 0.011767387390136719 time for calcul the mask position with numpy : 0.04436516761779785 nb_pixel_total : 84692 time to create 1 rle with old method : 0.09619784355163574 time for calcul the mask position with numpy : 0.04709315299987793 nb_pixel_total : 243103 time to create 1 rle with new method : 0.5376110076904297 time for calcul the mask position with numpy : 0.047840118408203125 nb_pixel_total : 9357 time to create 1 rle with old method : 0.010885000228881836 time for calcul the mask position with numpy : 0.04018211364746094 nb_pixel_total : 168003 time to create 1 rle with new method : 0.5804502964019775 time for calcul the mask position with numpy : 0.045866966247558594 nb_pixel_total : 110631 time to create 1 rle with old method : 0.12810397148132324 time for calcul the mask position with numpy : 0.04436016082763672 nb_pixel_total : 276298 time to create 1 rle with new method : 0.5972285270690918 time for calcul the mask position with numpy : 0.04101681709289551 nb_pixel_total : 22266 time to create 1 rle with old method : 0.025324344635009766 time for calcul the mask position with numpy : 0.04117894172668457 nb_pixel_total : 83381 time to create 1 rle with old method : 0.09875798225402832 time for calcul the mask position with numpy : 0.05011725425720215 nb_pixel_total : 96680 time to create 1 rle with old method : 0.11782312393188477 time for calcul the mask position with numpy : 0.05455422401428223 nb_pixel_total : 259114 time to create 1 rle with new method : 0.6856865882873535 time for calcul the mask position with numpy : 0.043259620666503906 nb_pixel_total : 156252 time to create 1 rle with new method : 0.6825804710388184 create new chi : 7.5945961475372314 time to delete rle : 0.001958131790161133 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 15829 TO DO : save crop sub photo not yet done ! save time : 0.9776618480682373 nb_obj : 10 nb_hashtags : 7 time to prepare the origin masks : 10.14539122581482 time for calcul the mask position with numpy : 0.4254910945892334 nb_pixel_total : 6957654 time to create 1 rle with new method : 0.6102828979492188 time for calcul the mask position with numpy : 0.04284977912902832 nb_pixel_total : 71942 time to create 1 rle with old method : 0.08304715156555176 time for calcul the mask position with numpy : 0.03190779685974121 nb_pixel_total : 7699 time to create 1 rle with old method : 0.010338068008422852 time for calcul the mask position with numpy : 0.029851198196411133 nb_pixel_total : 134466 time to create 1 rle with old method : 0.16446852684020996 time for calcul the mask position with numpy : 0.03539919853210449 nb_pixel_total : 255311 time to create 1 rle with new method : 0.7401812076568604 time for calcul the mask position with numpy : 0.03292083740234375 nb_pixel_total : 58676 time to create 1 rle with old method : 0.06742596626281738 time for calcul the mask position with numpy : 0.030668020248413086 nb_pixel_total : 147419 time to create 1 rle with old method : 0.1589221954345703 time for calcul the mask position with numpy : 0.029894113540649414 nb_pixel_total : 110023 time to create 1 rle with old method : 0.1348438262939453 time for calcul the mask position with numpy : 0.03372001647949219 nb_pixel_total : 275419 time to create 1 rle with new method : 0.8966996669769287 time for calcul the mask position with numpy : 0.03146982192993164 nb_pixel_total : 195355 time to create 1 rle with new method : 0.5059051513671875 time for calcul the mask position with numpy : 0.03017878532409668 nb_pixel_total : 80436 time to create 1 rle with old method : 0.0897364616394043 create new chi : 4.385688304901123 time to delete rle : 0.0015826225280761719 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 15075 TO DO : save crop sub photo not yet done ! save time : 0.8363101482391357 nb_obj : 10 nb_hashtags : 6 time to prepare the origin masks : 11.990314960479736 time for calcul the mask position with numpy : 0.3258206844329834 nb_pixel_total : 6029874 time to create 1 rle with new method : 2.3431851863861084 time for calcul the mask position with numpy : 0.026471853256225586 nb_pixel_total : 61260 time to create 1 rle with old method : 0.07020902633666992 time for calcul the mask position with numpy : 0.06678342819213867 nb_pixel_total : 242270 time to create 1 rle with new method : 1.5378625392913818 time for calcul the mask position with numpy : 0.05058789253234863 nb_pixel_total : 700054 time to create 1 rle with new method : 0.9132776260375977 time for calcul the mask position with numpy : 0.043065786361694336 nb_pixel_total : 12443 time to create 1 rle with old method : 0.014034748077392578 time for calcul the mask position with numpy : 0.04172468185424805 nb_pixel_total : 125338 time to create 1 rle with old method : 0.13938140869140625 time for calcul the mask position with numpy : 0.05083584785461426 nb_pixel_total : 50422 time to create 1 rle with old method : 0.06125831604003906 time for calcul the mask position with numpy : 0.049588918685913086 nb_pixel_total : 15451 time to create 1 rle with old method : 0.016181230545043945 time for calcul the mask position with numpy : 0.04710578918457031 nb_pixel_total : 75408 time to create 1 rle with old method : 0.08365035057067871 time for calcul the mask position with numpy : 0.04513812065124512 nb_pixel_total : 189057 time to create 1 rle with new method : 1.3538248538970947 time for calcul the mask position with numpy : 0.05093979835510254 nb_pixel_total : 792823 time to create 1 rle with new method : 0.7377216815948486 create new chi : 8.290276527404785 time to delete rle : 0.0022220611572265625 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 14326 TO DO : save crop sub photo not yet done ! save time : 0.7798705101013184 nb_obj : 10 nb_hashtags : 5 time to prepare the origin masks : 11.533722162246704 time for calcul the mask position with numpy : 0.5603947639465332 nb_pixel_total : 5763561 time to create 1 rle with new method : 1.0784590244293213 time for calcul the mask position with numpy : 0.02837991714477539 nb_pixel_total : 343588 time to create 1 rle with new method : 0.5941016674041748 time for calcul the mask position with numpy : 0.043894290924072266 nb_pixel_total : 173564 time to create 1 rle with new method : 1.0160102844238281 time for calcul the mask position with numpy : 0.04395771026611328 nb_pixel_total : 53734 time to create 1 rle with old method : 0.05913853645324707 time for calcul the mask position with numpy : 0.029064178466796875 nb_pixel_total : 360413 time to create 1 rle with new method : 1.220691442489624 time for calcul the mask position with numpy : 0.0476224422454834 nb_pixel_total : 198545 time to create 1 rle with new method : 1.0707285404205322 time for calcul the mask position with numpy : 0.04908323287963867 nb_pixel_total : 277061 time to create 1 rle with new method : 0.9791114330291748 time for calcul the mask position with numpy : 0.04861044883728027 nb_pixel_total : 63909 time to create 1 rle with old method : 0.07679414749145508 time for calcul the mask position with numpy : 0.0612943172454834 nb_pixel_total : 84069 time to create 1 rle with old method : 0.10162138938903809 time for calcul the mask position with numpy : 0.0844111442565918 nb_pixel_total : 960620 time to create 1 rle with new method : 1.082822322845459 time for calcul the mask position with numpy : 0.04755520820617676 nb_pixel_total : 15336 time to create 1 rle with old method : 0.02042675018310547 create new chi : 8.587286710739136 time to delete rle : 0.003198385238647461 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 15828 TO DO : save crop sub photo not yet done ! save time : 0.8752744197845459 map_output_result : {1337553906: (0.0, 'Should be the crop_list due to order', 0.0), 1337553903: (0.0, 'Should be the crop_list due to order', 0.0), 1337553899: (0.0, 'Should be the crop_list due to order', 0.0), 1337553895: (0.0, 'Should be the crop_list due to order', 0.0), 1337553886: (0.0, 'Should be the crop_list due to order', 0.0), 1337553884: (0.0, 'Should be the crop_list due to order', 0.0), 1337553863: (0.0, 'Should be the crop_list due to order', 0.0), 1337553860: (0.0, 'Should be the crop_list due to order', 0.0), 1337553857: (0.0, 'Should be the crop_list due to order', 0.0), 1337553852: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1337553906, 1337553903, 1337553899, 1337553895, 1337553886, 1337553884, 1337553863, 1337553860, 1337553857, 1337553852] Looping around the photos to save general results len do output : 10 /1337553906.Didn't retrieve data . /1337553903.Didn't retrieve data . /1337553899.Didn't retrieve data . /1337553895.Didn't retrieve data . /1337553886.Didn't retrieve data . /1337553884.Didn't retrieve data . /1337553863.Didn't retrieve data . /1337553860.Didn't retrieve data . /1337553857.Didn't retrieve data . /1337553852.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553906', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553903', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553899', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553895', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553886', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553884', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553863', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553860', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553857', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553852', None, None, None, None, None, '2585704') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013747453689575195 save_final save missing photos in datou_result : time spend for datou_step_exec : 155.3211328983307 time spend to save output : 0.014304637908935547 total time spend for step 5 : 155.33543753623962 step6:crop_condition Fri Feb 14 07:39:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 101 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! About to upload 10 photos upload in portfolio : 4869462 init cache_photo without model_param we have 10 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739515201_3340985 we have uploaded 10 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.5884766578674316 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 4869462 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739515214_3340985 we have uploaded 9 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.7513184547424316 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739515220_3340985 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.3201677799224854 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739515229_3340985 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.5044517517089844 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739515234_3340985 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.6635022163391113 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739515238_3340985 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6537113189697266 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1337553906, 1337553903, 1337553899, 1337553895, 1337553886, 1337553884, 1337553863, 1337553860, 1337553857, 1337553852] Looping around the photos to save general results len do output : 31 /1337559887Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559888Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559889Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559890Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559891Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559892Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559893Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559894Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559895Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559896Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559909Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559910Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559911Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559912Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559913Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559914Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559916Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559917Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559918Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559924Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559925Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559926Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559927Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559948Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559949Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559950Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559951Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559952Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559954Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559956Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337559960Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553906', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553903', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553899', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553895', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553886', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553884', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553863', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553860', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553857', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553852', None, None, None, None, None, '2585704') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 103 time used for this insertion : 0.036366939544677734 save_final save missing photos in datou_result : time spend for datou_step_exec : 52.145840883255005 time spend to save output : 0.03747725486755371 total time spend for step 6 : 52.18331813812256 step7:ventilate_hashtags_in_portfolio Fri Feb 14 07:40:45 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20544938 get user id for portfolio 20544938 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20544938 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Teint_Dans_La_Masse','environnement','kraft','papier','metal','plastique','flou','mal_croppe','Carton_brun','autre_refus','Carton_gris','cartonnette')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20544938 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Teint_Dans_La_Masse','environnement','kraft','papier','metal','plastique','flou','mal_croppe','Carton_brun','autre_refus','Carton_gris','cartonnette')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20544938 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Teint_Dans_La_Masse','environnement','kraft','papier','metal','plastique','flou','mal_croppe','Carton_brun','autre_refus','Carton_gris','cartonnette')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20545572,20545574,20545575,20545576,20545577,20545578,20545579,20545580,20545581,20545582,20545583,20545584?tags=Teint_Dans_La_Masse,environnement,kraft,papier,metal,plastique,flou,mal_croppe,Carton_brun,autre_refus,Carton_gris,cartonnette Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1337553906, 1337553903, 1337553899, 1337553895, 1337553886, 1337553884, 1337553863, 1337553860, 1337553857, 1337553852] Looping around the photos to save general results len do output : 1 /20544938. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553906', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553903', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553899', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553895', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553886', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553884', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553863', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553860', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553857', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553852', None, None, None, None, None, '2585704') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013766765594482422 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.628976583480835 time spend to save output : 0.014103174209594727 total time spend for step 7 : 1.6430797576904297 step8:final Fri Feb 14 07:40:47 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1337553906: ('0.12087122216415642',), 1337553903: ('0.12087122216415642',), 1337553899: ('0.12087122216415642',), 1337553895: ('0.12087122216415642',), 1337553886: ('0.12087122216415642',), 1337553884: ('0.12087122216415642',), 1337553863: ('0.12087122216415642',), 1337553860: ('0.12087122216415642',), 1337553857: ('0.12087122216415642',), 1337553852: ('0.12087122216415642',)} new output for save of step final : {1337553906: ('0.12087122216415642',), 1337553903: ('0.12087122216415642',), 1337553899: ('0.12087122216415642',), 1337553895: ('0.12087122216415642',), 1337553886: ('0.12087122216415642',), 1337553884: ('0.12087122216415642',), 1337553863: ('0.12087122216415642',), 1337553860: ('0.12087122216415642',), 1337553857: ('0.12087122216415642',), 1337553852: ('0.12087122216415642',)} [1337553906, 1337553903, 1337553899, 1337553895, 1337553886, 1337553884, 1337553863, 1337553860, 1337553857, 1337553852] Looping around the photos to save general results len do output : 10 /1337553906.Didn't retrieve data . /1337553903.Didn't retrieve data . /1337553899.Didn't retrieve data . /1337553895.Didn't retrieve data . /1337553886.Didn't retrieve data . /1337553884.Didn't retrieve data . /1337553863.Didn't retrieve data . /1337553860.Didn't retrieve data . /1337553857.Didn't retrieve data . /1337553852.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553906', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553903', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553899', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553895', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553886', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553884', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553863', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553860', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553857', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553852', None, None, None, None, None, '2585704') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013247251510620117 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.4948079586029053 time spend to save output : 0.013810873031616211 total time spend for step 8 : 0.5086188316345215 step9:velours_tree Fri Feb 14 07:40:47 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.03948378562927246 time spend to save output : 3.504753112792969e-05 total time spend for step 9 : 0.03951883316040039 step10:send_mail_cod Fri Feb 14 07:40:47 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20544938_14-02-2025_07_40_47.pdf 20545575 change filename to text .imagette205455751739515247 20545577 imagette205455771739515247 20545578 change filename to text .imagette205455781739515247 20545579 imagette205455791739515247 20545580 imagette205455801739515247 20545582 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette205455821739515247 20545584 change filename to text .change filename to text .change filename to text .change filename to text .imagette205455841739515248 20545572 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette205455721739515249 20545581 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette205455811739515249 20545583 imagette205455831739515250 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20544938 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20545572,20545574,20545575,20545576,20545577,20545578,20545579,20545580,20545581,20545582,20545583,20545584?tags=Teint_Dans_La_Masse,environnement,kraft,papier,metal,plastique,flou,mal_croppe,Carton_brun,autre_refus,Carton_gris,cartonnette your option no_mail is active, we will not send the real mail to your client args[1337553906] : ('0.12087122216415642',) no score found for photo 1337553906 apple ('0.12087122216415642',) We are sending mail with results at report@fotonower.com args[1337553903] : ('0.12087122216415642',) no score found for photo 1337553903 apple ('0.12087122216415642',) We are sending mail with results at report@fotonower.com args[1337553899] : ('0.12087122216415642',) no score found for photo 1337553899 apple ('0.12087122216415642',) We are sending mail with results at report@fotonower.com args[1337553895] : ('0.12087122216415642',) no score found for photo 1337553895 apple ('0.12087122216415642',) We are sending mail with results at report@fotonower.com args[1337553886] : ('0.12087122216415642',) no score found for photo 1337553886 apple ('0.12087122216415642',) We are sending mail with results at report@fotonower.com args[1337553884] : ('0.12087122216415642',) no score found for photo 1337553884 apple ('0.12087122216415642',) We are sending mail with results at report@fotonower.com args[1337553863] : ('0.12087122216415642',) no score found for photo 1337553863 apple ('0.12087122216415642',) We are sending mail with results at report@fotonower.com args[1337553860] : ('0.12087122216415642',) no score found for photo 1337553860 apple ('0.12087122216415642',) We are sending mail with results at report@fotonower.com args[1337553857] : ('0.12087122216415642',) no score found for photo 1337553857 apple ('0.12087122216415642',) We are sending mail with results at report@fotonower.com args[1337553852] : ('0.12087122216415642',) no score found for photo 1337553852 apple ('0.12087122216415642',) We are sending mail with results at report@fotonower.com refus_total : 0.12087122216415642 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20544938 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20544938_14-02-2025_07_40_47.pdf results_Auto_P20544938_14-02-2025_07_40_47.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20544938_14-02-2025_07_40_47.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20544938','results_Auto_P20544938_14-02-2025_07_40_47.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20544938_14-02-2025_07_40_47.pdf','pdf','','0.94','0.12087122216415642') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1337553906, 1337553903, 1337553899, 1337553895, 1337553886, 1337553884, 1337553863, 1337553860, 1337553857, 1337553852] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553906', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553903', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553899', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553895', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553886', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553884', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553863', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553860', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553857', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553852', None, None, None, None, None, '2585704') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.013829231262207031 save_final save missing photos in datou_result : time spend for datou_step_exec : 5.114310026168823 time spend to save output : 0.014210939407348633 total time spend for step 10 : 5.128520965576172 step11:split_time_score Fri Feb 14 07:40:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('06', 11),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 14022025 20544938 Nombre de photos uploadées : 11 / 23040 (0%) 14022025 20544938 Nombre de photos taguées (types de déchets): 0 / 11 (0%) 14022025 20544938 Nombre de photos taguées (volume) : 0 / 11 (0%) elapsed_time : load_data_split_time_score 1.9073486328125e-06 elapsed_time : order_list_meta_photo_and_scores 5.7220458984375e-06 ??????????? elapsed_time : fill_and_build_computed_from_old_data 0.0006067752838134766 elapsed_time : insert_dashboard_record_day_entry 0.025534868240356445 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.02707671464965672 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20544936_14-02-2025_07_36_41.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20544936 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20544936 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20544937 order by id desc limit 1 Qualite : 0.12087122216415642 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20544938_14-02-2025_07_40_47.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20544938 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20544938 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20544939 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20544940 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20544941 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'14022025': {'nb_upload': 11, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1337553906, 1337553903, 1337553899, 1337553895, 1337553886, 1337553884, 1337553863, 1337553860, 1337553857, 1337553852] Looping around the photos to save general results len do output : 1 /20544938Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553906', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553903', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553899', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553895', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553886', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553884', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553863', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553860', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553857', None, None, None, None, None, '2585704') ('3459', None, None, None, None, None, None, None, '2585704') ('3459', '20544938', '1337553852', None, None, None, None, None, '2585704') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.025524139404296875 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.3746118545532227 time spend to save output : 0.025748252868652344 total time spend for step 11 : 3.400360107421875 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 177.29user 174.51system 6:36.04elapsed 88%CPU (0avgtext+0avgdata 3982936maxresident)k 505480inputs+187880outputs (1510major+15655004minor)pagefaults 0swaps