python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1673409 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 18>10 please execute split_portfolio.py -i 20354367 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2567947'] with mtr_portfolio_ids : ['20354367'] and first list_photo_ids : [] new path : /proc/1673409/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.0862491130828857 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Sat Feb 8 01:38:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-08 01:38:33.148843: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-08 01:38:33.175249: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-08 01:38:33.177586: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f24d0000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-08 01:38:33.177643: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-08 01:38:33.181561: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-08 01:38:33.400971: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x116a62b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-08 01:38:33.401019: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-08 01:38:33.402380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-08 01:38:33.402772: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-08 01:38:33.405775: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-08 01:38:33.408736: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-08 01:38:33.409215: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-08 01:38:33.412430: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-08 01:38:33.413409: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-08 01:38:33.417765: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-08 01:38:33.419268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-08 01:38:33.419345: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-08 01:38:33.420143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-08 01:38:33.420158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-08 01:38:33.420167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-08 01:38:33.421431: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-08 01:38:33.677630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-08 01:38:33.677765: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-08 01:38:33.677788: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-08 01:38:33.677806: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-08 01:38:33.677825: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-08 01:38:33.677843: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-08 01:38:33.677860: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-08 01:38:33.677878: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-08 01:38:33.679437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-08 01:38:33.680830: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-08 01:38:33.680864: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-08 01:38:33.680883: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-08 01:38:33.680900: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-08 01:38:33.680916: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-08 01:38:33.680933: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-08 01:38:33.680950: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-08 01:38:33.680966: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-08 01:38:33.682488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-08 01:38:33.682528: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-08 01:38:33.682538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-08 01:38:33.682548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-08 01:38:33.684166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-08 01:38:41.032324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-08 01:38:41.201063: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 23 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 26 Detection mask done ! Trying to reset tf kernel 1673753 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5485 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10774 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.002167224884033203 nb_pixel_total : 94093 time to create 1 rle with old method : 0.09688973426818848 length of segment : 452 time for calcul the mask position with numpy : 0.0003440380096435547 nb_pixel_total : 18558 time to create 1 rle with old method : 0.019090890884399414 length of segment : 167 time for calcul the mask position with numpy : 7.104873657226562e-05 nb_pixel_total : 1443 time to create 1 rle with old method : 0.0016970634460449219 length of segment : 61 time for calcul the mask position with numpy : 0.003046274185180664 nb_pixel_total : 223527 time to create 1 rle with new method : 0.008228302001953125 length of segment : 595 time for calcul the mask position with numpy : 0.0005357265472412109 nb_pixel_total : 22204 time to create 1 rle with old method : 0.02408313751220703 length of segment : 269 time for calcul the mask position with numpy : 0.0004439353942871094 nb_pixel_total : 27537 time to create 1 rle with old method : 0.02859640121459961 length of segment : 227 time for calcul the mask position with numpy : 0.005456209182739258 nb_pixel_total : 434393 time to create 1 rle with new method : 0.015424489974975586 length of segment : 963 time for calcul the mask position with numpy : 0.0010213851928710938 nb_pixel_total : 50890 time to create 1 rle with old method : 0.054721832275390625 length of segment : 369 time for calcul the mask position with numpy : 0.0007245540618896484 nb_pixel_total : 44526 time to create 1 rle with old method : 0.048483848571777344 length of segment : 294 time for calcul the mask position with numpy : 0.001348257064819336 nb_pixel_total : 90198 time to create 1 rle with old method : 0.09773492813110352 length of segment : 370 time for calcul the mask position with numpy : 0.00029540061950683594 nb_pixel_total : 17643 time to create 1 rle with old method : 0.019181251525878906 length of segment : 172 time for calcul the mask position with numpy : 0.0016241073608398438 nb_pixel_total : 80272 time to create 1 rle with old method : 0.08458161354064941 length of segment : 441 time for calcul the mask position with numpy : 0.006167411804199219 nb_pixel_total : 308260 time to create 1 rle with new method : 0.020539045333862305 length of segment : 556 time for calcul the mask position with numpy : 8.916854858398438e-05 nb_pixel_total : 1744 time to create 1 rle with old method : 0.0020596981048583984 length of segment : 39 time for calcul the mask position with numpy : 0.002938508987426758 nb_pixel_total : 130409 time to create 1 rle with old method : 0.1289827823638916 length of segment : 602 time for calcul the mask position with numpy : 0.00796961784362793 nb_pixel_total : 431294 time to create 1 rle with new method : 0.02570366859436035 length of segment : 1130 time for calcul the mask position with numpy : 0.0003998279571533203 nb_pixel_total : 18690 time to create 1 rle with old method : 0.01928424835205078 length of segment : 192 time for calcul the mask position with numpy : 0.0032083988189697266 nb_pixel_total : 193753 time to create 1 rle with new method : 0.010236740112304688 length of segment : 946 time for calcul the mask position with numpy : 0.0001373291015625 nb_pixel_total : 3824 time to create 1 rle with old method : 0.004212379455566406 length of segment : 112 time for calcul the mask position with numpy : 0.0002560615539550781 nb_pixel_total : 12815 time to create 1 rle with old method : 0.013602256774902344 length of segment : 128 time for calcul the mask position with numpy : 0.004372596740722656 nb_pixel_total : 271437 time to create 1 rle with new method : 0.012038230895996094 length of segment : 624 time for calcul the mask position with numpy : 0.03609156608581543 nb_pixel_total : 1880391 time to create 1 rle with new method : 0.1182243824005127 length of segment : 2752 time for calcul the mask position with numpy : 0.0010519027709960938 nb_pixel_total : 57407 time to create 1 rle with old method : 0.06011962890625 length of segment : 330 time for calcul the mask position with numpy : 0.0008952617645263672 nb_pixel_total : 56352 time to create 1 rle with old method : 0.06053423881530762 length of segment : 210 time for calcul the mask position with numpy : 0.00016999244689941406 nb_pixel_total : 3786 time to create 1 rle with old method : 0.004130125045776367 length of segment : 119 time for calcul the mask position with numpy : 0.00037932395935058594 nb_pixel_total : 21205 time to create 1 rle with old method : 0.02208852767944336 length of segment : 186 time for calcul the mask position with numpy : 0.0008554458618164062 nb_pixel_total : 52494 time to create 1 rle with old method : 0.0528111457824707 length of segment : 445 time for calcul the mask position with numpy : 0.0012683868408203125 nb_pixel_total : 84985 time to create 1 rle with old method : 0.08523845672607422 length of segment : 421 time for calcul the mask position with numpy : 0.0017480850219726562 nb_pixel_total : 106640 time to create 1 rle with old method : 0.10802721977233887 length of segment : 569 time for calcul the mask position with numpy : 0.0003948211669921875 nb_pixel_total : 22366 time to create 1 rle with old method : 0.023169279098510742 length of segment : 181 time for calcul the mask position with numpy : 0.00023627281188964844 nb_pixel_total : 14746 time to create 1 rle with old method : 0.01562190055847168 length of segment : 108 time for calcul the mask position with numpy : 0.004970550537109375 nb_pixel_total : 350557 time to create 1 rle with new method : 0.012357711791992188 length of segment : 559 time for calcul the mask position with numpy : 0.08281540870666504 nb_pixel_total : 5103302 time to create 1 rle with new method : 0.144059419631958 length of segment : 3794 time for calcul the mask position with numpy : 0.0008480548858642578 nb_pixel_total : 33760 time to create 1 rle with old method : 0.03548765182495117 length of segment : 311 time for calcul the mask position with numpy : 0.01689434051513672 nb_pixel_total : 893463 time to create 1 rle with new method : 0.030066251754760742 length of segment : 1476 time for calcul the mask position with numpy : 0.00028896331787109375 nb_pixel_total : 6832 time to create 1 rle with old method : 0.007304191589355469 length of segment : 86 time for calcul the mask position with numpy : 0.07110953330993652 nb_pixel_total : 3113993 time to create 1 rle with new method : 0.13143658638000488 length of segment : 3610 time for calcul the mask position with numpy : 0.000110626220703125 nb_pixel_total : 3562 time to create 1 rle with old method : 0.0038597583770751953 length of segment : 53 time for calcul the mask position with numpy : 0.0010976791381835938 nb_pixel_total : 74411 time to create 1 rle with old method : 0.07543230056762695 length of segment : 376 time for calcul the mask position with numpy : 0.009171485900878906 nb_pixel_total : 603607 time to create 1 rle with new method : 0.024068355560302734 length of segment : 1104 time for calcul the mask position with numpy : 0.0029528141021728516 nb_pixel_total : 274015 time to create 1 rle with new method : 0.0066187381744384766 length of segment : 655 time for calcul the mask position with numpy : 9.822845458984375e-05 nb_pixel_total : 3136 time to create 1 rle with old method : 0.003499269485473633 length of segment : 98 time for calcul the mask position with numpy : 0.0003769397735595703 nb_pixel_total : 20631 time to create 1 rle with old method : 0.022304534912109375 length of segment : 320 time for calcul the mask position with numpy : 0.0002663135528564453 nb_pixel_total : 17947 time to create 1 rle with old method : 0.019855499267578125 length of segment : 160 time for calcul the mask position with numpy : 0.059868812561035156 nb_pixel_total : 3319987 time to create 1 rle with new method : 0.10790610313415527 length of segment : 3369 time for calcul the mask position with numpy : 0.004212141036987305 nb_pixel_total : 181433 time to create 1 rle with new method : 0.006880044937133789 length of segment : 716 time for calcul the mask position with numpy : 0.0005385875701904297 nb_pixel_total : 20877 time to create 1 rle with old method : 0.0215151309967041 length of segment : 190 time for calcul the mask position with numpy : 0.00045013427734375 nb_pixel_total : 19683 time to create 1 rle with old method : 0.020374536514282227 length of segment : 165 time for calcul the mask position with numpy : 0.0007860660552978516 nb_pixel_total : 25416 time to create 1 rle with old method : 0.026932954788208008 length of segment : 527 time for calcul the mask position with numpy : 0.0002803802490234375 nb_pixel_total : 5330 time to create 1 rle with old method : 0.0057566165924072266 length of segment : 120 time for calcul the mask position with numpy : 0.0012216567993164062 nb_pixel_total : 76294 time to create 1 rle with old method : 0.07650375366210938 length of segment : 442 time for calcul the mask position with numpy : 0.0013706684112548828 nb_pixel_total : 49434 time to create 1 rle with old method : 0.05217576026916504 length of segment : 423 time for calcul the mask position with numpy : 0.00022292137145996094 nb_pixel_total : 2637 time to create 1 rle with old method : 0.003055572509765625 length of segment : 106 time for calcul the mask position with numpy : 0.0009064674377441406 nb_pixel_total : 30703 time to create 1 rle with old method : 0.03371143341064453 length of segment : 343 time for calcul the mask position with numpy : 0.0001766681671142578 nb_pixel_total : 3131 time to create 1 rle with old method : 0.003550291061401367 length of segment : 61 time for calcul the mask position with numpy : 0.002675294876098633 nb_pixel_total : 81451 time to create 1 rle with old method : 0.08808469772338867 length of segment : 374 time for calcul the mask position with numpy : 0.008831501007080078 nb_pixel_total : 458057 time to create 1 rle with new method : 0.01609635353088379 length of segment : 870 time for calcul the mask position with numpy : 0.00022125244140625 nb_pixel_total : 4741 time to create 1 rle with old method : 0.005249500274658203 length of segment : 105 time for calcul the mask position with numpy : 0.00013875961303710938 nb_pixel_total : 1482 time to create 1 rle with old method : 0.0017316341400146484 length of segment : 53 time for calcul the mask position with numpy : 0.0025014877319335938 nb_pixel_total : 77055 time to create 1 rle with old method : 0.0799245834350586 length of segment : 393 time for calcul the mask position with numpy : 0.00025343894958496094 nb_pixel_total : 6031 time to create 1 rle with old method : 0.006678581237792969 length of segment : 65 time for calcul the mask position with numpy : 0.0006806850433349609 nb_pixel_total : 27747 time to create 1 rle with old method : 0.02928614616394043 length of segment : 175 time for calcul the mask position with numpy : 0.0004832744598388672 nb_pixel_total : 20089 time to create 1 rle with old method : 0.021617650985717773 length of segment : 174 time for calcul the mask position with numpy : 0.00046062469482421875 nb_pixel_total : 16761 time to create 1 rle with old method : 0.018190383911132812 length of segment : 158 time for calcul the mask position with numpy : 0.0019397735595703125 nb_pixel_total : 79874 time to create 1 rle with old method : 0.08461618423461914 length of segment : 306 time for calcul the mask position with numpy : 0.0015172958374023438 nb_pixel_total : 73809 time to create 1 rle with old method : 0.07808947563171387 length of segment : 205 time for calcul the mask position with numpy : 0.0008878707885742188 nb_pixel_total : 54829 time to create 1 rle with old method : 0.056807518005371094 length of segment : 221 time for calcul the mask position with numpy : 0.0002493858337402344 nb_pixel_total : 7221 time to create 1 rle with old method : 0.007500886917114258 length of segment : 115 time for calcul the mask position with numpy : 0.00044035911560058594 nb_pixel_total : 21900 time to create 1 rle with old method : 0.022365093231201172 length of segment : 150 time for calcul the mask position with numpy : 0.001878976821899414 nb_pixel_total : 62481 time to create 1 rle with old method : 0.06348633766174316 length of segment : 332 time for calcul the mask position with numpy : 0.0008111000061035156 nb_pixel_total : 13418 time to create 1 rle with old method : 0.01406550407409668 length of segment : 401 time for calcul the mask position with numpy : 0.00023508071899414062 nb_pixel_total : 5485 time to create 1 rle with old method : 0.005716085433959961 length of segment : 116 time for calcul the mask position with numpy : 0.001013040542602539 nb_pixel_total : 54216 time to create 1 rle with old method : 0.05416297912597656 length of segment : 305 time for calcul the mask position with numpy : 0.0017173290252685547 nb_pixel_total : 65353 time to create 1 rle with old method : 0.06752586364746094 length of segment : 315 time for calcul the mask position with numpy : 0.0003044605255126953 nb_pixel_total : 4809 time to create 1 rle with old method : 0.005412101745605469 length of segment : 114 time for calcul the mask position with numpy : 0.004556417465209961 nb_pixel_total : 193523 time to create 1 rle with new method : 0.007460832595825195 length of segment : 605 time for calcul the mask position with numpy : 0.0007979869842529297 nb_pixel_total : 27819 time to create 1 rle with old method : 0.028872251510620117 length of segment : 211 time for calcul the mask position with numpy : 0.0239870548248291 nb_pixel_total : 1265960 time to create 1 rle with new method : 0.04175519943237305 length of segment : 2091 time for calcul the mask position with numpy : 0.00034999847412109375 nb_pixel_total : 15972 time to create 1 rle with old method : 0.017714500427246094 length of segment : 139 time for calcul the mask position with numpy : 0.0001068115234375 nb_pixel_total : 997 time to create 1 rle with old method : 0.0010895729064941406 length of segment : 47 time for calcul the mask position with numpy : 0.00014781951904296875 nb_pixel_total : 2551 time to create 1 rle with old method : 0.0028657913208007812 length of segment : 77 time for calcul the mask position with numpy : 0.0009081363677978516 nb_pixel_total : 66071 time to create 1 rle with old method : 0.06895327568054199 length of segment : 333 time for calcul the mask position with numpy : 0.0017724037170410156 nb_pixel_total : 62709 time to create 1 rle with old method : 0.06591057777404785 length of segment : 310 time for calcul the mask position with numpy : 0.0010557174682617188 nb_pixel_total : 73865 time to create 1 rle with old method : 0.07520556449890137 length of segment : 342 time for calcul the mask position with numpy : 0.0019216537475585938 nb_pixel_total : 74327 time to create 1 rle with old method : 0.07861590385437012 length of segment : 455 time for calcul the mask position with numpy : 0.00014495849609375 nb_pixel_total : 2907 time to create 1 rle with old method : 0.0032422542572021484 length of segment : 45 time for calcul the mask position with numpy : 0.007976531982421875 nb_pixel_total : 506957 time to create 1 rle with new method : 0.017332792282104492 length of segment : 906 time for calcul the mask position with numpy : 0.00034689903259277344 nb_pixel_total : 13331 time to create 1 rle with old method : 0.014055728912353516 length of segment : 116 time for calcul the mask position with numpy : 0.003518342971801758 nb_pixel_total : 153498 time to create 1 rle with new method : 0.0042498111724853516 length of segment : 422 time for calcul the mask position with numpy : 0.00021123886108398438 nb_pixel_total : 4092 time to create 1 rle with old method : 0.004292964935302734 length of segment : 105 time for calcul the mask position with numpy : 0.0005941390991210938 nb_pixel_total : 19038 time to create 1 rle with old method : 0.021137714385986328 length of segment : 168 time for calcul the mask position with numpy : 0.0006909370422363281 nb_pixel_total : 17184 time to create 1 rle with old method : 0.018358707427978516 length of segment : 233 time for calcul the mask position with numpy : 0.0003819465637207031 nb_pixel_total : 22235 time to create 1 rle with old method : 0.02441716194152832 length of segment : 109 time for calcul the mask position with numpy : 0.0005924701690673828 nb_pixel_total : 15481 time to create 1 rle with old method : 0.01679372787475586 length of segment : 183 time for calcul the mask position with numpy : 0.00023937225341796875 nb_pixel_total : 10947 time to create 1 rle with old method : 0.011551141738891602 length of segment : 117 time for calcul the mask position with numpy : 0.0007114410400390625 nb_pixel_total : 21929 time to create 1 rle with old method : 0.023486852645874023 length of segment : 124 time for calcul the mask position with numpy : 0.003926515579223633 nb_pixel_total : 216722 time to create 1 rle with new method : 0.0062677860260009766 length of segment : 627 time for calcul the mask position with numpy : 0.0004901885986328125 nb_pixel_total : 10199 time to create 1 rle with old method : 0.010894536972045898 length of segment : 134 time for calcul the mask position with numpy : 0.0006854534149169922 nb_pixel_total : 20459 time to create 1 rle with old method : 0.020867586135864258 length of segment : 208 time for calcul the mask position with numpy : 0.0038938522338867188 nb_pixel_total : 121282 time to create 1 rle with old method : 0.12455368041992188 length of segment : 546 time for calcul the mask position with numpy : 0.00029277801513671875 nb_pixel_total : 5256 time to create 1 rle with old method : 0.005565643310546875 length of segment : 134 time for calcul the mask position with numpy : 0.0019690990447998047 nb_pixel_total : 71756 time to create 1 rle with old method : 0.07396817207336426 length of segment : 407 time for calcul the mask position with numpy : 0.0046977996826171875 nb_pixel_total : 179392 time to create 1 rle with new method : 0.006543397903442383 length of segment : 583 time for calcul the mask position with numpy : 0.0018749237060546875 nb_pixel_total : 62625 time to create 1 rle with old method : 0.06589698791503906 length of segment : 406 time for calcul the mask position with numpy : 0.0019488334655761719 nb_pixel_total : 61001 time to create 1 rle with old method : 0.06535744667053223 length of segment : 292 time for calcul the mask position with numpy : 0.00036025047302246094 nb_pixel_total : 7310 time to create 1 rle with old method : 0.008387327194213867 length of segment : 115 time for calcul the mask position with numpy : 0.0003120899200439453 nb_pixel_total : 5636 time to create 1 rle with old method : 0.006297588348388672 length of segment : 80 time for calcul the mask position with numpy : 0.007294654846191406 nb_pixel_total : 268867 time to create 1 rle with new method : 0.010404109954833984 length of segment : 630 time for calcul the mask position with numpy : 0.0019078254699707031 nb_pixel_total : 58994 time to create 1 rle with old method : 0.06271171569824219 length of segment : 470 time for calcul the mask position with numpy : 0.0006346702575683594 nb_pixel_total : 28285 time to create 1 rle with old method : 0.0308072566986084 length of segment : 297 time for calcul the mask position with numpy : 0.0002722740173339844 nb_pixel_total : 4600 time to create 1 rle with old method : 0.004994869232177734 length of segment : 100 time for calcul the mask position with numpy : 0.0017232894897460938 nb_pixel_total : 61260 time to create 1 rle with old method : 0.06376099586486816 length of segment : 659 time for calcul the mask position with numpy : 0.0026814937591552734 nb_pixel_total : 53819 time to create 1 rle with old method : 0.05815529823303223 length of segment : 402 time for calcul the mask position with numpy : 0.0011301040649414062 nb_pixel_total : 66478 time to create 1 rle with old method : 0.06773924827575684 length of segment : 321 time for calcul the mask position with numpy : 0.00021123886108398438 nb_pixel_total : 8080 time to create 1 rle with old method : 0.008626461029052734 length of segment : 144 time for calcul the mask position with numpy : 0.0028917789459228516 nb_pixel_total : 103952 time to create 1 rle with old method : 0.10533642768859863 length of segment : 434 time for calcul the mask position with numpy : 0.00023412704467773438 nb_pixel_total : 5038 time to create 1 rle with old method : 0.005396604537963867 length of segment : 92 time for calcul the mask position with numpy : 0.0003070831298828125 nb_pixel_total : 9180 time to create 1 rle with old method : 0.009834766387939453 length of segment : 152 time for calcul the mask position with numpy : 0.002882719039916992 nb_pixel_total : 98088 time to create 1 rle with old method : 0.1006772518157959 length of segment : 330 time spent for convertir_results : 14.058899402618408 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 119 chid ids of type : 3663 Number RLEs to save : 52742 save missing photos in datou_result : time spend for datou_step_exec : 45.49436330795288 time spend to save output : 5.431419610977173 total time spend for step 1 : 50.925782918930054 step2:crop_condition Sat Feb 8 01:39:21 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 119 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 34 About to insert : list_path_to_insert length 34 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335643385, 1335643382, 1335643378, 1335643377, 1335643374, 1335643160, 1335643149, 1335642928, 1335642927, 1335642905] Looping around the photos to save general results len do output : 67 /-3660975370Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975359Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975428Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975434Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975440Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975458Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975371Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975375Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975397Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975409Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975432Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975453Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975373Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975366Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975388Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975381Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975392Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975416Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975360Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975362Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975368Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975367Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975369Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975390Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975378Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975387Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975385Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975399Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975408Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975405Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975407Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975404Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975406Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975422Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975420Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975410Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975412Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975443Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975441Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975442Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975447Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975455Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975454Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975471Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975474Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975477Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975460Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975463Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975468Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975466Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975470Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975472Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975363Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975384Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975435Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975467Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975473Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975364Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975382Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975389Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975402Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975401Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975418Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975414Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975415Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975431Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3660975461Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643385', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643382', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643378', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643377', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643374', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643160', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643149', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642928', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642927', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642905', None, None, None, None, None, '2567947') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 211 time used for this insertion : 2.3036773204803467 save_final save missing photos in datou_result : time spend for datou_step_exec : 29.85670804977417 time spend to save output : 2.3065078258514404 total time spend for step 2 : 32.16321587562561 step3:thcl Sat Feb 8 01:39:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.006197214126586914 time to convert the images to numpy array : 0.05437898635864258 time to import caffe and check if the image exist : 0.009627342224121094 time to convert the images to numpy array : 0.06461644172668457 time to import caffe and check if the image exist : 0.00708317756652832 time to convert the images to numpy array : 0.07008552551269531 time to import caffe and check if the image exist : 0.006594419479370117 time to convert the images to numpy array : 0.07992219924926758 time to import caffe and check if the image exist : 0.009483575820922852 time to convert the images to numpy array : 0.07765007019042969 time to import caffe and check if the image exist : 0.011502504348754883 time to convert the images to numpy array : 0.07691764831542969 time to import caffe and check if the image exist : 0.0066759586334228516 time to convert the images to numpy array : 0.08605480194091797 time to import caffe and check if the image exist : 0.008154630661010742 time to convert the images to numpy array : 0.08680319786071777 time to import caffe and check if the image exist : 0.01109933853149414 time to convert the images to numpy array : 0.08912110328674316 time to import caffe and check if the image exist : 0.007934093475341797 time to convert the images to numpy array : 0.09706568717956543 total time to convert the images to numpy array : 0.35024333000183105 list photo_ids error: [] list photo_ids correct : [-3660975473, -3660975364, -3660975382, -3660975389, -3660975402, -3660975401, -3660975418, -3660975408, -3660975405, -3660975407, -3660975404, -3660975406, -3660975422, -3660975420, -3660975388, -3660975381, -3660975392, -3660975416, -3660975360, -3660975362, -3660975368, -3660975454, -3660975471, -3660975474, -3660975477, -3660975460, -3660975463, -3660975468, -3660975466, -3660975470, -3660975472, -3660975363, -3660975384, -3660975435, -3660975467, -3660975375, -3660975397, -3660975409, -3660975432, -3660975453, -3660975373, -3660975366, -3660975414, -3660975415, -3660975431, -3660975461, -3660975410, -3660975412, -3660975443, -3660975441, -3660975442, -3660975447, -3660975455, -3660975367, -3660975369, -3660975390, -3660975378, -3660975387, -3660975385, -3660975399, -3660975370, -3660975359, -3660975428, -3660975434, -3660975440, -3660975458, -3660975371] number of photos to traite : 67 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10998 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10776 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.6994438171386719 time used to do the prediction : 0.17388319969177246 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.7016820907592773 time used to do the prediction : 0.12685608863830566 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.5497207641601562e-05 save missing photos in datou_result : time spend for datou_step_exec : 8.576115846633911 time spend to save output : 0.0005118846893310547 total time spend for step 3 : 8.576627731323242 step4:merge_mask_thcl_custom Sat Feb 8 01:40:02 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 119 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1335643385, 1335643382, 1335643378, 1335643377, 1335643374, 1335643160, 1335643149, 1335642928, 1335642927, 1335642905] Looping around the photos to save general results len do output : 10 /1335643385Didn't retrieve data .Didn't retrieve data . /1335643382Didn't retrieve data .Didn't retrieve data . /1335643378Didn't retrieve data .Didn't retrieve data . /1335643377Didn't retrieve data .Didn't retrieve data . /1335643374Didn't retrieve data .Didn't retrieve data . /1335643160Didn't retrieve data .Didn't retrieve data . /1335643149Didn't retrieve data .Didn't retrieve data . /1335642928Didn't retrieve data .Didn't retrieve data . /1335642927Didn't retrieve data .Didn't retrieve data . /1335642905Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643385', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643382', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643378', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643377', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643374', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643160', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643149', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642928', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642927', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642905', None, None, None, None, None, '2567947') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01707315444946289 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.07247471809387207 time spend to save output : 0.017594099044799805 total time spend for step 4 : 0.09006881713867188 step5:rle_unique_nms_with_priority Sat Feb 8 01:40:02 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 17 nb_hashtags : 6 time to prepare the origin masks : 9.766098022460938 time for calcul the mask position with numpy : 0.49501824378967285 nb_pixel_total : 5062439 time to create 1 rle with new method : 0.5737960338592529 time for calcul the mask position with numpy : 0.01936483383178711 nb_pixel_total : 2218 time to create 1 rle with old method : 0.0024263858795166016 time for calcul the mask position with numpy : 0.0196378231048584 nb_pixel_total : 50819 time to create 1 rle with old method : 0.05218005180358887 time for calcul the mask position with numpy : 0.01970982551574707 nb_pixel_total : 22146 time to create 1 rle with old method : 0.022689104080200195 time for calcul the mask position with numpy : 0.020542621612548828 nb_pixel_total : 97045 time to create 1 rle with old method : 0.09830141067504883 time for calcul the mask position with numpy : 0.02282428741455078 nb_pixel_total : 307455 time to create 1 rle with new method : 0.5387852191925049 time for calcul the mask position with numpy : 0.020437240600585938 nb_pixel_total : 17643 time to create 1 rle with old method : 0.018384933471679688 time for calcul the mask position with numpy : 0.021030187606811523 nb_pixel_total : 90008 time to create 1 rle with old method : 0.09802746772766113 time for calcul the mask position with numpy : 0.025566577911376953 nb_pixel_total : 426273 time to create 1 rle with new method : 0.7134649753570557 time for calcul the mask position with numpy : 0.020771503448486328 nb_pixel_total : 27505 time to create 1 rle with old method : 0.0288693904876709 time for calcul the mask position with numpy : 0.02089238166809082 nb_pixel_total : 1723 time to create 1 rle with old method : 0.0019559860229492188 time for calcul the mask position with numpy : 0.021121501922607422 nb_pixel_total : 44479 time to create 1 rle with old method : 0.04623866081237793 time for calcul the mask position with numpy : 0.02390265464782715 nb_pixel_total : 93759 time to create 1 rle with old method : 0.09461069107055664 time for calcul the mask position with numpy : 0.02437138557434082 nb_pixel_total : 433860 time to create 1 rle with new method : 0.5495686531066895 time for calcul the mask position with numpy : 0.020131587982177734 nb_pixel_total : 18463 time to create 1 rle with old method : 0.019475460052490234 time for calcul the mask position with numpy : 0.02893543243408203 nb_pixel_total : 1436 time to create 1 rle with old method : 0.0016894340515136719 time for calcul the mask position with numpy : 0.03462934494018555 nb_pixel_total : 223073 time to create 1 rle with new method : 0.5551009178161621 time for calcul the mask position with numpy : 0.02937936782836914 nb_pixel_total : 129896 time to create 1 rle with old method : 0.1277775764465332 create new chi : 4.553194522857666 time to delete rle : 0.0137176513671875 batch 1 Loaded 18 chid ids of type : 3726 Number RLEs to save : 16236 TO DO : save crop sub photo not yet done ! save time : 1.2347469329833984 nb_obj : 14 nb_hashtags : 5 time to prepare the origin masks : 6.465178489685059 time for calcul the mask position with numpy : 0.39760875701904297 nb_pixel_total : 4286895 time to create 1 rle with new method : 1.1252632141113281 time for calcul the mask position with numpy : 0.038774967193603516 nb_pixel_total : 349928 time to create 1 rle with new method : 1.536388874053955 time for calcul the mask position with numpy : 0.021975040435791016 nb_pixel_total : 355 time to create 1 rle with old method : 0.0009210109710693359 time for calcul the mask position with numpy : 0.023732423782348633 nb_pixel_total : 12789 time to create 1 rle with old method : 0.015637636184692383 time for calcul the mask position with numpy : 0.03702545166015625 nb_pixel_total : 2626 time to create 1 rle with old method : 0.003220796585083008 time for calcul the mask position with numpy : 0.03835940361022949 nb_pixel_total : 22088 time to create 1 rle with old method : 0.023345470428466797 time for calcul the mask position with numpy : 0.038495779037475586 nb_pixel_total : 5876 time to create 1 rle with old method : 0.009714126586914062 time for calcul the mask position with numpy : 0.04021620750427246 nb_pixel_total : 52656 time to create 1 rle with old method : 0.05660533905029297 time for calcul the mask position with numpy : 0.03441166877746582 nb_pixel_total : 14689 time to create 1 rle with old method : 0.015825748443603516 time for calcul the mask position with numpy : 0.04055476188659668 nb_pixel_total : 3773 time to create 1 rle with old method : 0.004164695739746094 time for calcul the mask position with numpy : 0.03628945350646973 nb_pixel_total : 108430 time to create 1 rle with old method : 0.11647725105285645 time for calcul the mask position with numpy : 0.05776643753051758 nb_pixel_total : 1859800 time to create 1 rle with new method : 0.9427676200866699 time for calcul the mask position with numpy : 0.03880429267883301 nb_pixel_total : 270387 time to create 1 rle with new method : 0.4821133613586426 time for calcul the mask position with numpy : 0.044909000396728516 nb_pixel_total : 3778 time to create 1 rle with old method : 0.007155656814575195 time for calcul the mask position with numpy : 0.05298972129821777 nb_pixel_total : 56170 time to create 1 rle with old method : 0.062283992767333984 create new chi : 5.491073369979858 time to delete rle : 0.002015829086303711 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 14220 TO DO : save crop sub photo not yet done ! save time : 1.0536117553710938 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 1.6269116401672363 time for calcul the mask position with numpy : 0.028847932815551758 nb_pixel_total : 1047036 time to create 1 rle with new method : 0.13838553428649902 time for calcul the mask position with numpy : 0.02030324935913086 nb_pixel_total : 110453 time to create 1 rle with old method : 0.11567354202270508 time for calcul the mask position with numpy : 0.07020688056945801 nb_pixel_total : 5892751 time to create 1 rle with new method : 0.4863317012786865 create new chi : 0.8999319076538086 time to delete rle : 0.0006766319274902344 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 7685 TO DO : save crop sub photo not yet done ! save time : 0.5228009223937988 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 5.314904451370239 time for calcul the mask position with numpy : 0.13940119743347168 nb_pixel_total : 2992583 time to create 1 rle with new method : 1.5291805267333984 time for calcul the mask position with numpy : 0.022428035736083984 nb_pixel_total : 11964 time to create 1 rle with old method : 0.013135910034179688 time for calcul the mask position with numpy : 0.024606704711914062 nb_pixel_total : 66669 time to create 1 rle with old method : 0.07217693328857422 time for calcul the mask position with numpy : 0.02505016326904297 nb_pixel_total : 260177 time to create 1 rle with new method : 0.542640209197998 time for calcul the mask position with numpy : 0.023651599884033203 nb_pixel_total : 3120 time to create 1 rle with old method : 0.003548145294189453 time for calcul the mask position with numpy : 0.02960944175720215 nb_pixel_total : 573142 time to create 1 rle with new method : 0.7699074745178223 time for calcul the mask position with numpy : 0.026394128799438477 nb_pixel_total : 3551 time to create 1 rle with old method : 0.003983259201049805 time for calcul the mask position with numpy : 0.12003302574157715 nb_pixel_total : 3111631 time to create 1 rle with new method : 0.3442525863647461 time for calcul the mask position with numpy : 0.024402618408203125 nb_pixel_total : 20603 time to create 1 rle with old method : 0.02313852310180664 time for calcul the mask position with numpy : 0.023972272872924805 nb_pixel_total : 6800 time to create 1 rle with old method : 0.011063098907470703 create new chi : 3.8737621307373047 time to delete rle : 0.002388477325439453 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 14449 TO DO : save crop sub photo not yet done ! save time : 1.3907365798950195 nb_obj : 7 nb_hashtags : 6 time to prepare the origin masks : 3.785796880722046 time for calcul the mask position with numpy : 0.09093236923217773 nb_pixel_total : 3424406 time to create 1 rle with new method : 0.31035304069519043 time for calcul the mask position with numpy : 0.021949291229248047 nb_pixel_total : 76268 time to create 1 rle with old method : 0.08451080322265625 time for calcul the mask position with numpy : 0.019836902618408203 nb_pixel_total : 2611 time to create 1 rle with old method : 0.002950429916381836 time for calcul the mask position with numpy : 0.01978588104248047 nb_pixel_total : 19641 time to create 1 rle with old method : 0.021046161651611328 time for calcul the mask position with numpy : 0.0218966007232666 nb_pixel_total : 180308 time to create 1 rle with new method : 0.6013085842132568 time for calcul the mask position with numpy : 0.03426861763000488 nb_pixel_total : 20827 time to create 1 rle with old method : 0.023924827575683594 time for calcul the mask position with numpy : 0.10122537612915039 nb_pixel_total : 3300968 time to create 1 rle with new method : 0.3514738082885742 time for calcul the mask position with numpy : 0.03728795051574707 nb_pixel_total : 25211 time to create 1 rle with old method : 0.027702808380126953 create new chi : 1.8472142219543457 time to delete rle : 0.001321554183959961 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 13035 TO DO : save crop sub photo not yet done ! save time : 1.064157247543335 nb_obj : 13 nb_hashtags : 4 time to prepare the origin masks : 6.868612051010132 time for calcul the mask position with numpy : 0.3046278953552246 nb_pixel_total : 6271757 time to create 1 rle with new method : 0.33908963203430176 time for calcul the mask position with numpy : 0.028520584106445312 nb_pixel_total : 4731 time to create 1 rle with old method : 0.005945920944213867 time for calcul the mask position with numpy : 0.03772306442260742 nb_pixel_total : 76896 time to create 1 rle with old method : 0.09287619590759277 time for calcul the mask position with numpy : 0.04111766815185547 nb_pixel_total : 81325 time to create 1 rle with old method : 0.09059786796569824 time for calcul the mask position with numpy : 0.03809094429016113 nb_pixel_total : 19995 time to create 1 rle with old method : 0.020419836044311523 time for calcul the mask position with numpy : 0.03883504867553711 nb_pixel_total : 5984 time to create 1 rle with old method : 0.006380319595336914 time for calcul the mask position with numpy : 0.038572072982788086 nb_pixel_total : 1467 time to create 1 rle with old method : 0.0016133785247802734 time for calcul the mask position with numpy : 0.03913283348083496 nb_pixel_total : 3106 time to create 1 rle with old method : 0.0033292770385742188 time for calcul the mask position with numpy : 0.04006457328796387 nb_pixel_total : 49437 time to create 1 rle with old method : 0.05269980430603027 time for calcul the mask position with numpy : 0.03858017921447754 nb_pixel_total : 2623 time to create 1 rle with old method : 0.0030655860900878906 time for calcul the mask position with numpy : 0.03900575637817383 nb_pixel_total : 27603 time to create 1 rle with old method : 0.029855966567993164 time for calcul the mask position with numpy : 0.025351524353027344 nb_pixel_total : 30547 time to create 1 rle with old method : 0.03276395797729492 time for calcul the mask position with numpy : 0.03142547607421875 nb_pixel_total : 458103 time to create 1 rle with new method : 0.4275786876678467 time for calcul the mask position with numpy : 0.032036781311035156 nb_pixel_total : 16666 time to create 1 rle with old method : 0.01862621307373047 create new chi : 1.9756691455841064 time to delete rle : 0.0013327598571777344 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 8768 TO DO : save crop sub photo not yet done ! save time : 1.1355819702148438 nb_obj : 9 nb_hashtags : 3 time to prepare the origin masks : 5.169653654098511 time for calcul the mask position with numpy : 0.2658650875091553 nb_pixel_total : 6745921 time to create 1 rle with new method : 0.33286309242248535 time for calcul the mask position with numpy : 0.023859262466430664 nb_pixel_total : 62194 time to create 1 rle with old method : 0.06686091423034668 time for calcul the mask position with numpy : 0.028937339782714844 nb_pixel_total : 5463 time to create 1 rle with old method : 0.006328582763671875 time for calcul the mask position with numpy : 0.04273366928100586 nb_pixel_total : 12246 time to create 1 rle with old method : 0.013254165649414062 time for calcul the mask position with numpy : 0.025321006774902344 nb_pixel_total : 13415 time to create 1 rle with old method : 0.01410222053527832 time for calcul the mask position with numpy : 0.03618764877319336 nb_pixel_total : 21828 time to create 1 rle with old method : 0.0221707820892334 time for calcul the mask position with numpy : 0.0503847599029541 nb_pixel_total : 7188 time to create 1 rle with old method : 0.00856924057006836 time for calcul the mask position with numpy : 0.04179716110229492 nb_pixel_total : 54273 time to create 1 rle with old method : 0.05540323257446289 time for calcul the mask position with numpy : 0.062154293060302734 nb_pixel_total : 54391 time to create 1 rle with old method : 0.05875539779663086 time for calcul the mask position with numpy : 0.0485234260559082 nb_pixel_total : 73321 time to create 1 rle with old method : 0.07745766639709473 create new chi : 1.3180010318756104 time to delete rle : 0.00092315673828125 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 6343 TO DO : save crop sub photo not yet done ! save time : 0.6811883449554443 nb_obj : 13 nb_hashtags : 3 time to prepare the origin masks : 5.105952024459839 time for calcul the mask position with numpy : 0.1389329433441162 nb_pixel_total : 4984823 time to create 1 rle with new method : 0.37705183029174805 time for calcul the mask position with numpy : 0.02559947967529297 nb_pixel_total : 921 time to create 1 rle with old method : 0.0012698173522949219 time for calcul the mask position with numpy : 0.02646780014038086 nb_pixel_total : 73630 time to create 1 rle with old method : 0.07889151573181152 time for calcul the mask position with numpy : 0.027074575424194336 nb_pixel_total : 27824 time to create 1 rle with old method : 0.033262014389038086 time for calcul the mask position with numpy : 0.02920699119567871 nb_pixel_total : 251371 time to create 1 rle with new method : 0.41739964485168457 time for calcul the mask position with numpy : 0.0341792106628418 nb_pixel_total : 193401 time to create 1 rle with new method : 0.2966620922088623 time for calcul the mask position with numpy : 0.024403095245361328 nb_pixel_total : 2878 time to create 1 rle with old method : 0.0032606124877929688 time for calcul the mask position with numpy : 0.021953105926513672 nb_pixel_total : 2521 time to create 1 rle with old method : 0.0029506683349609375 time for calcul the mask position with numpy : 0.021793603897094727 nb_pixel_total : 989 time to create 1 rle with old method : 0.0013966560363769531 time for calcul the mask position with numpy : 0.023441791534423828 nb_pixel_total : 15874 time to create 1 rle with old method : 0.01791071891784668 time for calcul the mask position with numpy : 0.038665056228637695 nb_pixel_total : 1354461 time to create 1 rle with new method : 0.4044201374053955 time for calcul the mask position with numpy : 0.02087116241455078 nb_pixel_total : 4788 time to create 1 rle with old method : 0.005566835403442383 time for calcul the mask position with numpy : 0.02198195457458496 nb_pixel_total : 62541 time to create 1 rle with old method : 0.0764763355255127 time for calcul the mask position with numpy : 0.0219271183013916 nb_pixel_total : 74218 time to create 1 rle with old method : 0.07899594306945801 create new chi : 2.3762776851654053 time to delete rle : 0.001102447509765625 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 11644 TO DO : save crop sub photo not yet done ! save time : 8.204486846923828 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 4.91625452041626 time for calcul the mask position with numpy : 0.6349358558654785 nb_pixel_total : 6582128 time to create 1 rle with new method : 1.0003604888916016 time for calcul the mask position with numpy : 0.030451059341430664 nb_pixel_total : 214315 time to create 1 rle with new method : 0.7888376712799072 time for calcul the mask position with numpy : 0.04564714431762695 nb_pixel_total : 10882 time to create 1 rle with old method : 0.019582748413085938 time for calcul the mask position with numpy : 0.04214906692504883 nb_pixel_total : 21749 time to create 1 rle with old method : 0.03449869155883789 time for calcul the mask position with numpy : 0.0601963996887207 nb_pixel_total : 14706 time to create 1 rle with old method : 0.027970314025878906 time for calcul the mask position with numpy : 0.05183720588684082 nb_pixel_total : 17112 time to create 1 rle with old method : 0.02092742919921875 time for calcul the mask position with numpy : 0.04409599304199219 nb_pixel_total : 18851 time to create 1 rle with old method : 0.02042245864868164 time for calcul the mask position with numpy : 0.04544520378112793 nb_pixel_total : 153138 time to create 1 rle with new method : 0.7628402709960938 time for calcul the mask position with numpy : 0.043947696685791016 nb_pixel_total : 4085 time to create 1 rle with old method : 0.004625797271728516 time for calcul the mask position with numpy : 0.0342869758605957 nb_pixel_total : 13274 time to create 1 rle with old method : 0.016525983810424805 create new chi : 3.84734845161438 time to delete rle : 0.000993490219116211 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 6030 TO DO : save crop sub photo not yet done ! save time : 0.4390721321105957 nb_obj : 22 nb_hashtags : 5 time to prepare the origin masks : 14.04390811920166 time for calcul the mask position with numpy : 0.4299461841583252 nb_pixel_total : 5778569 time to create 1 rle with new method : 0.7065274715423584 time for calcul the mask position with numpy : 0.030708789825439453 nb_pixel_total : 53370 time to create 1 rle with old method : 0.058081865310668945 time for calcul the mask position with numpy : 0.033975839614868164 nb_pixel_total : 28162 time to create 1 rle with old method : 0.0308992862701416 time for calcul the mask position with numpy : 0.03912973403930664 nb_pixel_total : 60911 time to create 1 rle with old method : 0.08112144470214844 time for calcul the mask position with numpy : 0.0395817756652832 nb_pixel_total : 268522 time to create 1 rle with new method : 0.593172550201416 time for calcul the mask position with numpy : 0.046202659606933594 nb_pixel_total : 63440 time to create 1 rle with old method : 0.07131218910217285 time for calcul the mask position with numpy : 0.03560447692871094 nb_pixel_total : 9134 time to create 1 rle with old method : 0.010879278182983398 time for calcul the mask position with numpy : 0.034197330474853516 nb_pixel_total : 5011 time to create 1 rle with old method : 0.005857944488525391 time for calcul the mask position with numpy : 0.03611254692077637 nb_pixel_total : 8073 time to create 1 rle with old method : 0.009133100509643555 time for calcul the mask position with numpy : 0.058701515197753906 nb_pixel_total : 103272 time to create 1 rle with old method : 0.11117982864379883 time for calcul the mask position with numpy : 0.05290865898132324 nb_pixel_total : 178834 time to create 1 rle with new method : 0.5366489887237549 time for calcul the mask position with numpy : 0.04833102226257324 nb_pixel_total : 4570 time to create 1 rle with old method : 0.005346775054931641 time for calcul the mask position with numpy : 0.05673861503601074 nb_pixel_total : 5594 time to create 1 rle with old method : 0.006083488464355469 time for calcul the mask position with numpy : 0.04876828193664551 nb_pixel_total : 71348 time to create 1 rle with old method : 0.07680749893188477 time for calcul the mask position with numpy : 0.04694080352783203 nb_pixel_total : 7243 time to create 1 rle with old method : 0.008289813995361328 time for calcul the mask position with numpy : 0.05036449432373047 nb_pixel_total : 6086 time to create 1 rle with old method : 0.007199764251708984 time for calcul the mask position with numpy : 0.04993009567260742 nb_pixel_total : 69143 time to create 1 rle with old method : 0.07456588745117188 time for calcul the mask position with numpy : 0.04826545715332031 nb_pixel_total : 5235 time to create 1 rle with old method : 0.005764484405517578 time for calcul the mask position with numpy : 0.04441499710083008 nb_pixel_total : 20378 time to create 1 rle with old method : 0.023122549057006836 time for calcul the mask position with numpy : 0.028458356857299805 nb_pixel_total : 10128 time to create 1 rle with old method : 0.0111541748046875 time for calcul the mask position with numpy : 0.034430503845214844 nb_pixel_total : 121084 time to create 1 rle with old method : 0.13100266456604004 time for calcul the mask position with numpy : 0.032506465911865234 nb_pixel_total : 74453 time to create 1 rle with old method : 0.07902693748474121 time for calcul the mask position with numpy : 0.030735015869140625 nb_pixel_total : 97680 time to create 1 rle with old method : 0.10375666618347168 create new chi : 4.230471849441528 time to delete rle : 0.002228975296020508 batch 1 Loaded 23 chid ids of type : 3726 Number RLEs to save : 14770 TO DO : save crop sub photo not yet done ! save time : 1.3049941062927246 map_output_result : {1335643385: (0.0, 'Should be the crop_list due to order', 0.0), 1335643382: (0.0, 'Should be the crop_list due to order', 0.0), 1335643378: (0.0, 'Should be the crop_list due to order', 0.0), 1335643377: (0.0, 'Should be the crop_list due to order', 0.0), 1335643374: (0.0, 'Should be the crop_list due to order', 0.0), 1335643160: (0.0, 'Should be the crop_list due to order', 0.0), 1335643149: (0.0, 'Should be the crop_list due to order', 0.0), 1335642928: (0.0, 'Should be the crop_list due to order', 0.0), 1335642927: (0.0, 'Should be the crop_list due to order', 0.0), 1335642905: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1335643385, 1335643382, 1335643378, 1335643377, 1335643374, 1335643160, 1335643149, 1335642928, 1335642927, 1335642905] Looping around the photos to save general results len do output : 10 /1335643385.Didn't retrieve data . /1335643382.Didn't retrieve data . /1335643378.Didn't retrieve data . /1335643377.Didn't retrieve data . /1335643374.Didn't retrieve data . /1335643160.Didn't retrieve data . /1335643149.Didn't retrieve data . /1335642928.Didn't retrieve data . /1335642927.Didn't retrieve data . /1335642905.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643385', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643382', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643378', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643377', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643374', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643160', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643149', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642928', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642927', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642905', None, None, None, None, None, '2567947') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014907121658325195 save_final save missing photos in datou_result : time spend for datou_step_exec : 111.60342121124268 time spend to save output : 0.1435542106628418 total time spend for step 5 : 111.74697542190552 step6:crop_condition Sat Feb 8 01:41:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 125 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738975316_1673409 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 6.683412790298462 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738975325_1673409 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.011003017425537 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738975327_1673409 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6675119400024414 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738975329_1673409 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6072263717651367 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738975331_1673409 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.1886539459228516 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738975337_1673409 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.8349828720092773 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335643385, 1335643382, 1335643378, 1335643377, 1335643374, 1335643160, 1335643149, 1335642928, 1335642927, 1335642905] Looping around the photos to save general results len do output : 16 /1335666182Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666184Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666185Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666186Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666187Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666188Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666190Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666191Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666192Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666193Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666194Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666197Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666198Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666199Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666200Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335666201Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643385', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643382', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643378', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643377', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643374', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643160', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643149', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642928', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642927', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642905', None, None, None, None, None, '2567947') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 58 time used for this insertion : 0.1252422332763672 save_final save missing photos in datou_result : time spend for datou_step_exec : 30.84281086921692 time spend to save output : 0.12601804733276367 total time spend for step 6 : 30.968828916549683 step7:ventilate_hashtags_in_portfolio Sat Feb 8 01:42:24 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20354367 get user id for portfolio 20354367 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20354367 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_brun','plastique','autre_refus','mal_croppe','cartonnette','environnement','metal','kraft','Teint_Dans_La_Masse','papier','flou','Carton_gris')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20354367 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_brun','plastique','autre_refus','mal_croppe','cartonnette','environnement','metal','kraft','Teint_Dans_La_Masse','papier','flou','Carton_gris')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20354367 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_brun','plastique','autre_refus','mal_croppe','cartonnette','environnement','metal','kraft','Teint_Dans_La_Masse','papier','flou','Carton_gris')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20354449,20354450,20354451,20354452,20354453,20354454,20354455,20354456,20354457,20354458,20354459,20354460?tags=Carton_brun,plastique,autre_refus,mal_croppe,cartonnette,environnement,metal,kraft,Teint_Dans_La_Masse,papier,flou,Carton_gris Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1335643385, 1335643382, 1335643378, 1335643377, 1335643374, 1335643160, 1335643149, 1335642928, 1335642927, 1335642905] Looping around the photos to save general results len do output : 1 /20354367. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643385', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643382', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643378', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643377', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643374', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643160', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643149', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642928', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642927', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642905', None, None, None, None, None, '2567947') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.018238544464111328 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.0933384895324707 time spend to save output : 0.018634557723999023 total time spend for step 7 : 2.1119730472564697 step8:final Sat Feb 8 01:42:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1335643385: ('0.050100097939186815',), 1335643382: ('0.050100097939186815',), 1335643378: ('0.050100097939186815',), 1335643377: ('0.050100097939186815',), 1335643374: ('0.050100097939186815',), 1335643160: ('0.050100097939186815',), 1335643149: ('0.050100097939186815',), 1335642928: ('0.050100097939186815',), 1335642927: ('0.050100097939186815',), 1335642905: ('0.050100097939186815',)} new output for save of step final : {1335643385: ('0.050100097939186815',), 1335643382: ('0.050100097939186815',), 1335643378: ('0.050100097939186815',), 1335643377: ('0.050100097939186815',), 1335643374: ('0.050100097939186815',), 1335643160: ('0.050100097939186815',), 1335643149: ('0.050100097939186815',), 1335642928: ('0.050100097939186815',), 1335642927: ('0.050100097939186815',), 1335642905: ('0.050100097939186815',)} [1335643385, 1335643382, 1335643378, 1335643377, 1335643374, 1335643160, 1335643149, 1335642928, 1335642927, 1335642905] Looping around the photos to save general results len do output : 10 /1335643385.Didn't retrieve data . /1335643382.Didn't retrieve data . /1335643378.Didn't retrieve data . /1335643377.Didn't retrieve data . /1335643374.Didn't retrieve data . /1335643160.Didn't retrieve data . /1335643149.Didn't retrieve data . /1335642928.Didn't retrieve data . /1335642927.Didn't retrieve data . /1335642905.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643385', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643382', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643378', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643377', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643374', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643160', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643149', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642928', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642927', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642905', None, None, None, None, None, '2567947') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01236414909362793 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11439990997314453 time spend to save output : 0.012928962707519531 total time spend for step 8 : 0.12732887268066406 step9:velours_tree Sat Feb 8 01:42:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.07699203491210938 time spend to save output : 8.511543273925781e-05 total time spend for step 9 : 0.07707715034484863 step10:send_mail_cod Sat Feb 8 01:42:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20354367_08-02-2025_01_42_27.pdf 20354450 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203544501738975347 20354451 change filename to text .change filename to text .imagette203544511738975347 20354452 imagette203544521738975347 20354453 change filename to text .imagette203544531738975347 20354455 imagette203544551738975347 20354456 imagette203544561738975347 20354459 imagette203544591738975347 20354449 change filename to text .change filename to text .change filename to text .change filename to text .imagette203544491738975347 20354457 change filename to text .change filename to text .imagette203544571738975348 20354460 change filename to text .change filename to text .imagette203544601738975348 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20354367 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20354449,20354450,20354451,20354452,20354453,20354454,20354455,20354456,20354457,20354458,20354459,20354460?tags=Carton_brun,plastique,autre_refus,mal_croppe,cartonnette,environnement,metal,kraft,Teint_Dans_La_Masse,papier,flou,Carton_gris your option no_mail is active, we will not send the real mail to your client args[1335643385] : ('0.050100097939186815',) no score found for photo 1335643385 apple ('0.050100097939186815',) We are sending mail with results at report@fotonower.com args[1335643382] : ('0.050100097939186815',) no score found for photo 1335643382 apple ('0.050100097939186815',) We are sending mail with results at report@fotonower.com args[1335643378] : ('0.050100097939186815',) no score found for photo 1335643378 apple ('0.050100097939186815',) We are sending mail with results at report@fotonower.com args[1335643377] : ('0.050100097939186815',) no score found for photo 1335643377 apple ('0.050100097939186815',) We are sending mail with results at report@fotonower.com args[1335643374] : ('0.050100097939186815',) no score found for photo 1335643374 apple ('0.050100097939186815',) We are sending mail with results at report@fotonower.com args[1335643160] : ('0.050100097939186815',) no score found for photo 1335643160 apple ('0.050100097939186815',) We are sending mail with results at report@fotonower.com args[1335643149] : ('0.050100097939186815',) no score found for photo 1335643149 apple ('0.050100097939186815',) We are sending mail with results at report@fotonower.com args[1335642928] : ('0.050100097939186815',) no score found for photo 1335642928 apple ('0.050100097939186815',) We are sending mail with results at report@fotonower.com args[1335642927] : ('0.050100097939186815',) no score found for photo 1335642927 apple ('0.050100097939186815',) We are sending mail with results at report@fotonower.com args[1335642905] : ('0.050100097939186815',) no score found for photo 1335642905 apple ('0.050100097939186815',) We are sending mail with results at report@fotonower.com refus_total : 0.050100097939186815 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20354367 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1335642648,1335643160,1335643149,1335642927,1335642905,1335642903,1335642795,1335642790,1335642749,1335642744,1335642928,1335642719,1335642450,1335643374,1335643377,1335643378,1335643382,1335643385) Found this number of photos: 18 begin to download photo : 1335642648 begin to download photo : 1335642905 begin to download photo : 1335642749 begin to download photo : 1335642450 begin to download photo : 1335643382 download finish for photo 1335642450 begin to download photo : 1335643374 download finish for photo 1335643382 begin to download photo : 1335643385 download finish for photo 1335642648 begin to download photo : 1335643160 download finish for photo 1335642905 begin to download photo : 1335642903 download finish for photo 1335642749 begin to download photo : 1335642744 download finish for photo 1335643374 begin to download photo : 1335643377 download finish for photo 1335643385 download finish for photo 1335643160 begin to download photo : 1335643149 download finish for photo 1335642744 begin to download photo : 1335642928 download finish for photo 1335642903 begin to download photo : 1335642795 download finish for photo 1335643377 begin to download photo : 1335643378 download finish for photo 1335643149 begin to download photo : 1335642927 download finish for photo 1335642928 begin to download photo : 1335642719 download finish for photo 1335642795 begin to download photo : 1335642790 download finish for photo 1335643378 download finish for photo 1335642927 download finish for photo 1335642719 download finish for photo 1335642790 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20354367_08-02-2025_01_42_27.pdf results_Auto_P20354367_08-02-2025_01_42_27.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20354367_08-02-2025_01_42_27.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20354367','results_Auto_P20354367_08-02-2025_01_42_27.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20354367_08-02-2025_01_42_27.pdf','pdf','','0.59','0.050100097939186815') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1335643385, 1335643382, 1335643378, 1335643377, 1335643374, 1335643160, 1335643149, 1335642928, 1335642927, 1335642905] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643385', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643382', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643378', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643377', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643374', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643160', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643149', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642928', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642927', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642905', None, None, None, None, None, '2567947') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.013623237609863281 save_final save missing photos in datou_result : time spend for datou_step_exec : 5.158819198608398 time spend to save output : 0.01392054557800293 total time spend for step 10 : 5.172739744186401 step11:split_time_score Sat Feb 8 01:42:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('16', 18),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 07022025 20354367 Nombre de photos uploadées : 18 / 23040 (0%) 07022025 20354367 Nombre de photos taguées (types de déchets): 0 / 18 (0%) 07022025 20354367 Nombre de photos taguées (volume) : 0 / 18 (0%) elapsed_time : load_data_split_time_score 2.86102294921875e-06 elapsed_time : order_list_meta_photo_and_scores 7.867813110351562e-06 ?????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.00113677978515625 elapsed_time : insert_dashboard_record_day_entry 0.0276491641998291 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354353 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354354 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354355 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354356 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354357 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354358 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354359 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354360 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354361 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351772 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351773 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351774 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351775 order by id desc limit 1 Qualite : 0.09932132178283122 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351776_07-02-2025_22_24_39.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351776 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351776 AND mptpi.`type`=3726 To do Qualite : 0.2385312928452032 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351777_07-02-2025_23_09_50.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351777 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351777 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351778 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351779 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351782 order by id desc limit 1 Qualite : 0.1972755360101216 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351807_07-02-2025_23_47_16.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351807 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351807 AND mptpi.`type`=3594 To do Qualite : 0.37949999486949176 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351814_07-02-2025_23_20_42.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351814 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351814 AND mptpi.`type`=4200 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351815 order by id desc limit 1 Qualite : 0.07767031081515724 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351816_08-02-2025_01_09_15.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351816 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11524 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11533 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11532 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11528 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11528 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11531 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11531 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11578 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11578 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11527 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11526 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11526 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11535 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11527 doesn't seem to be define in the database( WARNING : type of input 3 of step 11526 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11524 doesn't seem to be define in the database( WARNING : type of input 2 of step 11528 doesn't seem to be define in the database( WARNING : output 1 of step 11524 have datatype=2 whereas input 1 of step 11531 have datatype=7 WARNING : type of output 2 of step 11531 doesn't seem to be define in the database( WARNING : type of input 1 of step 11525 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11527 have datatype=10 whereas input 3 of step 11534 have datatype=6 WARNING : type of input 2 of step 11578 doesn't seem to be define in the database( WARNING : output 1 of step 11525 have datatype=7 whereas input 2 of step 11578 have datatype=None WARNING : type of output 3 of step 11578 doesn't seem to be define in the database( WARNING : type of input 1 of step 11527 doesn't seem to be define in the database( WARNING : output 0 of step 11527 have datatype=10 whereas input 0 of step 11584 have datatype=18 WARNING : type of input 5 of step 11534 doesn't seem to be define in the database( WARNING : output 0 of step 11584 have datatype=11 whereas input 5 of step 11534 have datatype=None WARNING : type of output 1 of step 11533 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : type of output 1 of step 11532 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : output 0 of step 11531 have datatype=1 whereas input 0 of step 11525 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351816 AND mptpi.`type`=4211 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351817 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354362 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354363 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354364 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354365 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354366 order by id desc limit 1 Qualite : 0.050100097939186815 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20354367_08-02-2025_01_42_27.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354367 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20354367 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354368 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354369 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354370 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354371 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354372 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354373 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354374 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20354375 order by id desc limit 1 Qualite : 0.4211175932107529 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351861_08-02-2025_01_32_39.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351861 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351861 AND mptpi.`type`=4200 To do Qualite : 0.08701264816490367 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351862_07-02-2025_22_27_08.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351862 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351862 AND mptpi.`type`=3726 To do Qualite : 0.18687183807817992 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351863_07-02-2025_22_32_02.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351863 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351863 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351864 order by id desc limit 1 Qualite : 0.03599672022473725 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351865_07-02-2025_23_21_30.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351865 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351865 AND mptpi.`type`=4207 To do Qualite : 0.11103449586161805 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20351866_08-02-2025_01_40_07.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20351866 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20351866 AND mptpi.`type`=4207 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'07022025': {'nb_upload': 18, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1335643385, 1335643382, 1335643378, 1335643377, 1335643374, 1335643160, 1335643149, 1335642928, 1335642927, 1335642905] Looping around the photos to save general results len do output : 1 /20354367Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643385', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643382', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643378', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643377', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643374', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643160', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335643149', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642928', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642927', None, None, None, None, None, '2567947') ('3459', None, None, None, None, None, None, None, '2567947') ('3459', '20354367', '1335642905', None, None, None, None, None, '2567947') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013979196548461914 save_final save missing photos in datou_result : time spend for datou_step_exec : 26.06685161590576 time spend to save output : 0.014200210571289062 total time spend for step 11 : 26.08105182647705 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 122.60user 86.40system 4:35.31elapsed 75%CPU (0avgtext+0avgdata 4145456maxresident)k 343600inputs+101712outputs (14557major+9741887minor)pagefaults 0swaps