python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1754908 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2647393'] with mtr_portfolio_ids : ['21136611'] and first list_photo_ids : [] new path : /proc/1754908/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , BFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 8 ; length of list_pids : 8 ; length of list_args : 8 time to download the photos : 2.7174232006073 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Mar 7 08:52:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10372 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-07 08:52:34.081507: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-07 08:52:34.107218: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-07 08:52:34.109760: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f74a8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-07 08:52:34.109832: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-07 08:52:34.114395: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-07 08:52:34.363980: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xf6f7ff0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-07 08:52:34.364042: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-07 08:52:34.365744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 08:52:34.366580: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 08:52:34.372449: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 08:52:34.375845: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 08:52:34.377099: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 08:52:34.379900: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 08:52:34.380986: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 08:52:34.386760: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 08:52:34.388252: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 08:52:34.388320: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 08:52:34.389140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-07 08:52:34.389158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-07 08:52:34.389169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-07 08:52:34.390688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9607 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-07 08:52:34.788072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 08:52:34.788177: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 08:52:34.788207: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 08:52:34.788227: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 08:52:34.788244: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 08:52:34.788260: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 08:52:34.788276: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 08:52:34.788293: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 08:52:34.789627: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 08:52:34.791034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 08:52:34.791106: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 08:52:34.791123: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 08:52:34.791137: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 08:52:34.791151: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 08:52:34.791165: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 08:52:34.791178: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 08:52:34.791193: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 08:52:34.792486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 08:52:34.792533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-07 08:52:34.792545: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-07 08:52:34.792555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-07 08:52:34.793949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9607 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-07 08:52:46.051271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 08:52:46.266058: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 8 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 141.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 3.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 142.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 1.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 147.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 140.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 17 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 149.35000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 148.85000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 17 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 3.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 145.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 15 Detection mask done ! Trying to reset tf kernel 1755373 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5083 tf kernel not reseted sub process len(results) : 8 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 8 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10372 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.3185420036315918 nb_pixel_total : 1985410 time to create 1 rle with new method : 0.19993114471435547 length of segment : 2154 time for calcul the mask position with numpy : 0.005094766616821289 nb_pixel_total : 256595 time to create 1 rle with new method : 0.014097929000854492 length of segment : 479 time for calcul the mask position with numpy : 0.0017747879028320312 nb_pixel_total : 93552 time to create 1 rle with old method : 0.10125255584716797 length of segment : 799 time for calcul the mask position with numpy : 0.0019683837890625 nb_pixel_total : 85310 time to create 1 rle with old method : 0.09924626350402832 length of segment : 299 time for calcul the mask position with numpy : 0.005098581314086914 nb_pixel_total : 266317 time to create 1 rle with new method : 0.009813308715820312 length of segment : 481 time for calcul the mask position with numpy : 0.007630348205566406 nb_pixel_total : 488163 time to create 1 rle with new method : 0.020677804946899414 length of segment : 689 time for calcul the mask position with numpy : 0.028609752655029297 nb_pixel_total : 1555044 time to create 1 rle with new method : 0.13524103164672852 length of segment : 2177 time for calcul the mask position with numpy : 0.0048503875732421875 nb_pixel_total : 172145 time to create 1 rle with new method : 0.009900569915771484 length of segment : 583 time for calcul the mask position with numpy : 0.0005915164947509766 nb_pixel_total : 15132 time to create 1 rle with old method : 0.01778864860534668 length of segment : 127 time for calcul the mask position with numpy : 0.012202739715576172 nb_pixel_total : 391098 time to create 1 rle with new method : 0.019358158111572266 length of segment : 842 time for calcul the mask position with numpy : 0.0006029605865478516 nb_pixel_total : 14840 time to create 1 rle with old method : 0.018070459365844727 length of segment : 158 time for calcul the mask position with numpy : 0.001611471176147461 nb_pixel_total : 23389 time to create 1 rle with old method : 0.028714656829833984 length of segment : 308 time for calcul the mask position with numpy : 0.004630565643310547 nb_pixel_total : 228890 time to create 1 rle with new method : 0.008609771728515625 length of segment : 651 time for calcul the mask position with numpy : 0.0007579326629638672 nb_pixel_total : 34296 time to create 1 rle with old method : 0.03817892074584961 length of segment : 227 time for calcul the mask position with numpy : 0.002959012985229492 nb_pixel_total : 170599 time to create 1 rle with new method : 0.0053517818450927734 length of segment : 373 time for calcul the mask position with numpy : 0.0012960433959960938 nb_pixel_total : 50142 time to create 1 rle with old method : 0.057093143463134766 length of segment : 347 time for calcul the mask position with numpy : 0.033417463302612305 nb_pixel_total : 1816341 time to create 1 rle with new method : 0.21034455299377441 length of segment : 2510 time for calcul the mask position with numpy : 0.00086212158203125 nb_pixel_total : 35224 time to create 1 rle with old method : 0.04081153869628906 length of segment : 253 time for calcul the mask position with numpy : 0.01802992820739746 nb_pixel_total : 802608 time to create 1 rle with new method : 0.41852331161499023 length of segment : 1007 time for calcul the mask position with numpy : 0.0013020038604736328 nb_pixel_total : 41924 time to create 1 rle with old method : 0.047640085220336914 length of segment : 318 time for calcul the mask position with numpy : 0.022949934005737305 nb_pixel_total : 938652 time to create 1 rle with new method : 0.31520986557006836 length of segment : 1943 time for calcul the mask position with numpy : 0.0005195140838623047 nb_pixel_total : 15110 time to create 1 rle with old method : 0.01775193214416504 length of segment : 200 time for calcul the mask position with numpy : 0.0030007362365722656 nb_pixel_total : 156123 time to create 1 rle with new method : 0.007146120071411133 length of segment : 1159 time for calcul the mask position with numpy : 0.006398677825927734 nb_pixel_total : 379111 time to create 1 rle with new method : 0.016289472579956055 length of segment : 1434 time for calcul the mask position with numpy : 0.0010859966278076172 nb_pixel_total : 57599 time to create 1 rle with old method : 0.06250238418579102 length of segment : 335 time for calcul the mask position with numpy : 0.003945350646972656 nb_pixel_total : 198839 time to create 1 rle with new method : 0.007319450378417969 length of segment : 686 time for calcul the mask position with numpy : 0.005524635314941406 nb_pixel_total : 292403 time to create 1 rle with new method : 0.011435270309448242 length of segment : 1011 time for calcul the mask position with numpy : 0.0006010532379150391 nb_pixel_total : 32941 time to create 1 rle with old method : 0.03756904602050781 length of segment : 179 time for calcul the mask position with numpy : 0.003641366958618164 nb_pixel_total : 150816 time to create 1 rle with new method : 0.004673480987548828 length of segment : 602 time for calcul the mask position with numpy : 0.003153562545776367 nb_pixel_total : 154268 time to create 1 rle with new method : 0.009382247924804688 length of segment : 600 time for calcul the mask position with numpy : 0.00039196014404296875 nb_pixel_total : 20738 time to create 1 rle with old method : 0.025969266891479492 length of segment : 152 time for calcul the mask position with numpy : 0.0036776065826416016 nb_pixel_total : 191454 time to create 1 rle with new method : 0.007987260818481445 length of segment : 642 time for calcul the mask position with numpy : 0.0025899410247802734 nb_pixel_total : 143756 time to create 1 rle with old method : 0.15868258476257324 length of segment : 431 time for calcul the mask position with numpy : 0.0007674694061279297 nb_pixel_total : 33260 time to create 1 rle with old method : 0.04460263252258301 length of segment : 269 time for calcul the mask position with numpy : 0.010926008224487305 nb_pixel_total : 456627 time to create 1 rle with new method : 0.022958040237426758 length of segment : 1310 time for calcul the mask position with numpy : 0.00742340087890625 nb_pixel_total : 421762 time to create 1 rle with new method : 0.016800880432128906 length of segment : 707 time for calcul the mask position with numpy : 0.0026106834411621094 nb_pixel_total : 112318 time to create 1 rle with old method : 0.13096070289611816 length of segment : 876 time for calcul the mask position with numpy : 0.0006766319274902344 nb_pixel_total : 18081 time to create 1 rle with old method : 0.02054309844970703 length of segment : 264 time for calcul the mask position with numpy : 0.00308990478515625 nb_pixel_total : 182960 time to create 1 rle with new method : 0.008721113204956055 length of segment : 528 time for calcul the mask position with numpy : 0.0073244571685791016 nb_pixel_total : 361512 time to create 1 rle with new method : 0.015913009643554688 length of segment : 972 time for calcul the mask position with numpy : 0.00028228759765625 nb_pixel_total : 10046 time to create 1 rle with old method : 0.012315750122070312 length of segment : 99 time for calcul the mask position with numpy : 0.011565208435058594 nb_pixel_total : 443203 time to create 1 rle with new method : 0.018259286880493164 length of segment : 737 time for calcul the mask position with numpy : 0.0017118453979492188 nb_pixel_total : 69546 time to create 1 rle with old method : 0.07574343681335449 length of segment : 362 time for calcul the mask position with numpy : 0.002624988555908203 nb_pixel_total : 115703 time to create 1 rle with old method : 0.13333868980407715 length of segment : 502 time for calcul the mask position with numpy : 0.002565145492553711 nb_pixel_total : 81868 time to create 1 rle with old method : 0.0918121337890625 length of segment : 391 time for calcul the mask position with numpy : 0.002215862274169922 nb_pixel_total : 69369 time to create 1 rle with old method : 0.07922887802124023 length of segment : 416 time for calcul the mask position with numpy : 0.006854057312011719 nb_pixel_total : 243051 time to create 1 rle with new method : 0.01283121109008789 length of segment : 752 time for calcul the mask position with numpy : 0.005022287368774414 nb_pixel_total : 229556 time to create 1 rle with new method : 0.009048938751220703 length of segment : 562 time for calcul the mask position with numpy : 0.006745815277099609 nb_pixel_total : 304602 time to create 1 rle with new method : 0.011377573013305664 length of segment : 989 time for calcul the mask position with numpy : 0.0005757808685302734 nb_pixel_total : 18387 time to create 1 rle with old method : 0.020647525787353516 length of segment : 165 time for calcul the mask position with numpy : 0.00038623809814453125 nb_pixel_total : 15170 time to create 1 rle with old method : 0.019919157028198242 length of segment : 166 time for calcul the mask position with numpy : 0.008359193801879883 nb_pixel_total : 365569 time to create 1 rle with new method : 0.015635013580322266 length of segment : 1071 time for calcul the mask position with numpy : 0.0079193115234375 nb_pixel_total : 416465 time to create 1 rle with new method : 0.017041444778442383 length of segment : 906 time for calcul the mask position with numpy : 0.0014100074768066406 nb_pixel_total : 57204 time to create 1 rle with old method : 0.0765538215637207 length of segment : 364 time for calcul the mask position with numpy : 0.0009129047393798828 nb_pixel_total : 34015 time to create 1 rle with old method : 0.04060530662536621 length of segment : 157 time for calcul the mask position with numpy : 0.0038089752197265625 nb_pixel_total : 136300 time to create 1 rle with old method : 0.17730164527893066 length of segment : 669 time for calcul the mask position with numpy : 0.005389213562011719 nb_pixel_total : 240073 time to create 1 rle with new method : 0.008229255676269531 length of segment : 751 time for calcul the mask position with numpy : 0.0021800994873046875 nb_pixel_total : 85799 time to create 1 rle with old method : 0.0950307846069336 length of segment : 233 time for calcul the mask position with numpy : 0.0004949569702148438 nb_pixel_total : 14298 time to create 1 rle with old method : 0.01793217658996582 length of segment : 128 time for calcul the mask position with numpy : 0.0024352073669433594 nb_pixel_total : 118243 time to create 1 rle with old method : 0.1494607925415039 length of segment : 402 time for calcul the mask position with numpy : 0.0012805461883544922 nb_pixel_total : 86982 time to create 1 rle with old method : 0.10456514358520508 length of segment : 234 time for calcul the mask position with numpy : 0.0013589859008789062 nb_pixel_total : 50540 time to create 1 rle with old method : 0.05921792984008789 length of segment : 304 time for calcul the mask position with numpy : 0.0030105113983154297 nb_pixel_total : 110131 time to create 1 rle with old method : 0.13065671920776367 length of segment : 391 time for calcul the mask position with numpy : 0.004361152648925781 nb_pixel_total : 162643 time to create 1 rle with new method : 0.007444858551025391 length of segment : 1045 time for calcul the mask position with numpy : 0.009548187255859375 nb_pixel_total : 503117 time to create 1 rle with new method : 0.01502847671508789 length of segment : 1122 time for calcul the mask position with numpy : 0.014136314392089844 nb_pixel_total : 699601 time to create 1 rle with new method : 0.04208803176879883 length of segment : 988 time for calcul the mask position with numpy : 0.006658077239990234 nb_pixel_total : 338781 time to create 1 rle with new method : 0.013738393783569336 length of segment : 570 time for calcul the mask position with numpy : 0.00642704963684082 nb_pixel_total : 373320 time to create 1 rle with new method : 0.011835336685180664 length of segment : 690 time for calcul the mask position with numpy : 0.012967109680175781 nb_pixel_total : 729817 time to create 1 rle with new method : 0.038643598556518555 length of segment : 979 time for calcul the mask position with numpy : 0.005628824234008789 nb_pixel_total : 218043 time to create 1 rle with new method : 0.009618759155273438 length of segment : 532 time for calcul the mask position with numpy : 0.0019850730895996094 nb_pixel_total : 88074 time to create 1 rle with old method : 0.10288548469543457 length of segment : 325 time for calcul the mask position with numpy : 0.008069515228271484 nb_pixel_total : 465778 time to create 1 rle with new method : 0.015364646911621094 length of segment : 788 time for calcul the mask position with numpy : 0.009400606155395508 nb_pixel_total : 423515 time to create 1 rle with new method : 0.015550374984741211 length of segment : 1212 time for calcul the mask position with numpy : 0.011044025421142578 nb_pixel_total : 404645 time to create 1 rle with new method : 0.019372940063476562 length of segment : 920 time for calcul the mask position with numpy : 0.001657247543334961 nb_pixel_total : 53275 time to create 1 rle with old method : 0.06266427040100098 length of segment : 367 time for calcul the mask position with numpy : 0.0027692317962646484 nb_pixel_total : 170043 time to create 1 rle with new method : 0.0054569244384765625 length of segment : 713 time for calcul the mask position with numpy : 0.0014920234680175781 nb_pixel_total : 101208 time to create 1 rle with old method : 0.11275458335876465 length of segment : 415 time for calcul the mask position with numpy : 0.003314971923828125 nb_pixel_total : 142279 time to create 1 rle with old method : 0.16245818138122559 length of segment : 949 time spent for convertir_results : 15.217903852462769 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 78 chid ids of type : 3663 Number RLEs to save : 51448 save missing photos in datou_result : time spend for datou_step_exec : 66.70862674713135 time spend to save output : 4.20708966255188 total time spend for step 1 : 70.91571640968323 step2:crop_condition Fri Mar 7 08:53:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 8 ! batch 1 Loaded 78 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 14 About to insert : list_path_to_insert length 14 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 25 About to insert : list_path_to_insert length 25 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1327679562, 1327679550, 1327679546, 1327679541, 1327679536, 1327679521, 1327679518, 1327679517] Looping around the photos to save general results len do output : 62 /-3697038037Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038055Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038058Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038056Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038066Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038067Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038068Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038095Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038042Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038051Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038050Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038046Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038062Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038074Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038076Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038069Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038083Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038088Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038087Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038082Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038100Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038105Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038047Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038064Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038070Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038078Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038073Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038081Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038098Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038110Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038054Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038075Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038092Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038094Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038089Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038093Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038036Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038038Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038040Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038043Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038044Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038039Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038049Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038048Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038053Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038059Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038063Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038060Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038086Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038091Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038097Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038102Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038101Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038096Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038099Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038103Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038107Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038106Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038111Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038108Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038104Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697038079Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679562', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679550', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679546', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679541', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679536', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679521', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679518', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679517', None, None, None, None, None, '2647393') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 194 time used for this insertion : 0.029750347137451172 save_final save missing photos in datou_result : time spend for datou_step_exec : 75.94475388526917 time spend to save output : 0.03185319900512695 total time spend for step 2 : 75.97660708427429 step3:thcl Fri Mar 7 08:54:58 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.006407022476196289 time to convert the images to numpy array : 4.291534423828125e-06 time to import caffe and check if the image exist : 0.010914087295532227 time to convert the images to numpy array : 0.5489881038665771 time to import caffe and check if the image exist : 0.010392427444458008 time to convert the images to numpy array : 1.6449244022369385 time to import caffe and check if the image exist : 0.011495590209960938 time to convert the images to numpy array : 1.6911849975585938 time to import caffe and check if the image exist : 0.011425018310546875 time to convert the images to numpy array : 1.8272814750671387 time to import caffe and check if the image exist : 0.009592533111572266 time to convert the images to numpy array : 1.8445677757263184 time to import caffe and check if the image exist : 0.010117053985595703 time to convert the images to numpy array : 2.0382959842681885 time to import caffe and check if the image exist : 0.006991386413574219 time to convert the images to numpy array : 2.060051202774048 time to import caffe and check if the image exist : 0.014773845672607422 time to convert the images to numpy array : 2.342198610305786 time to import caffe and check if the image exist : 0.01296544075012207 time to convert the images to numpy array : 2.3652637004852295 total time to convert the images to numpy array : 2.715920925140381 list photo_ids error: [] list photo_ids correct : [-3697038098, -3697038110, -3697038054, -3697038075, -3697038092, -3697038094, -3697038089, -3697038093, -3697038036, -3697038038, -3697038040, -3697038043, -3697038044, -3697038039, -3697038107, -3697038106, -3697038111, -3697038108, -3697038104, -3697038079, -3697038049, -3697038048, -3697038053, -3697038059, -3697038063, -3697038060, -3697038086, -3697038105, -3697038047, -3697038064, -3697038070, -3697038078, -3697038073, -3697038081, -3697038091, -3697038097, -3697038102, -3697038101, -3697038096, -3697038099, -3697038103, -3697038037, -3697038055, -3697038058, -3697038056, -3697038066, -3697038067, -3697038068, -3697038076, -3697038069, -3697038083, -3697038088, -3697038087, -3697038082, -3697038100, -3697038095, -3697038042, -3697038051, -3697038050, -3697038046, -3697038062, -3697038074] number of photos to traite : 62 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5304 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5085 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 2.7833197116851807 time used to do the prediction : 0.18065214157104492 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10372 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 3.191838026046753 time used to do the prediction : 0.13222146034240723 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.288818359375e-05 save missing photos in datou_result : time spend for datou_step_exec : 15.906092166900635 time spend to save output : 0.0005285739898681641 total time spend for step 3 : 15.906620740890503 step4:merge_mask_thcl_custom Fri Mar 7 08:55:14 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 78 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1327679562, 1327679550, 1327679546, 1327679541, 1327679536, 1327679521, 1327679518, 1327679517] Looping around the photos to save general results len do output : 8 /1327679562Didn't retrieve data .Didn't retrieve data . /1327679550Didn't retrieve data .Didn't retrieve data . /1327679546Didn't retrieve data .Didn't retrieve data . /1327679541Didn't retrieve data .Didn't retrieve data . /1327679536Didn't retrieve data .Didn't retrieve data . /1327679521Didn't retrieve data .Didn't retrieve data . /1327679518Didn't retrieve data .Didn't retrieve data . /1327679517Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679562', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679550', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679546', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679541', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679536', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679521', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679518', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679517', None, None, None, None, None, '2647393') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 24 time used for this insertion : 0.014598369598388672 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.1082160472869873 time spend to save output : 0.015096426010131836 total time spend for step 4 : 0.12331247329711914 step5:rle_unique_nms_with_priority Fri Mar 7 08:55:14 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 4 nb_hashtags : 3 time to prepare the origin masks : 5.014623403549194 time for calcul the mask position with numpy : 0.5626664161682129 nb_pixel_total : 5952134 time to create 1 rle with new method : 1.0072524547576904 time for calcul the mask position with numpy : 0.035124778747558594 nb_pixel_total : 92912 time to create 1 rle with old method : 0.10305237770080566 time for calcul the mask position with numpy : 0.023734569549560547 nb_pixel_total : 84129 time to create 1 rle with old method : 0.08673262596130371 time for calcul the mask position with numpy : 0.024803876876831055 nb_pixel_total : 176360 time to create 1 rle with new method : 0.7687442302703857 time for calcul the mask position with numpy : 0.17455697059631348 nb_pixel_total : 1988865 time to create 1 rle with new method : 0.6544933319091797 create new chi : 3.534921407699585 time to delete rle : 0.017852306365966797 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 9223 TO DO : save crop sub photo not yet done ! save time : 0.5236616134643555 nb_obj : 6 nb_hashtags : 4 time to prepare the origin masks : 6.2776710987091064 time for calcul the mask position with numpy : 0.44459986686706543 nb_pixel_total : 5783224 time to create 1 rle with new method : 2.851970911026001 time for calcul the mask position with numpy : 0.027680158615112305 nb_pixel_total : 6468 time to create 1 rle with old method : 0.008515119552612305 time for calcul the mask position with numpy : 0.03657817840576172 nb_pixel_total : 246139 time to create 1 rle with new method : 1.399681568145752 time for calcul the mask position with numpy : 0.027286767959594727 nb_pixel_total : 15075 time to create 1 rle with old method : 0.020200490951538086 time for calcul the mask position with numpy : 0.0460357666015625 nb_pixel_total : 1364802 time to create 1 rle with new method : 1.1665356159210205 time for calcul the mask position with numpy : 0.030894994735717773 nb_pixel_total : 391105 time to create 1 rle with new method : 0.7969620227813721 time for calcul the mask position with numpy : 0.044402360916137695 nb_pixel_total : 487587 time to create 1 rle with new method : 0.7324280738830566 create new chi : 7.794182300567627 time to delete rle : 0.0010750293731689453 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 9570 TO DO : save crop sub photo not yet done ! save time : 0.5563864707946777 nb_obj : 6 nb_hashtags : 4 time to prepare the origin masks : 8.23628568649292 time for calcul the mask position with numpy : 0.2973308563232422 nb_pixel_total : 6029123 time to create 1 rle with new method : 0.9375998973846436 time for calcul the mask position with numpy : 0.026853084564208984 nb_pixel_total : 34228 time to create 1 rle with old method : 0.03743267059326172 time for calcul the mask position with numpy : 0.8494117259979248 nb_pixel_total : 1758945 time to create 1 rle with new method : 0.5316293239593506 time for calcul the mask position with numpy : 0.04617190361022949 nb_pixel_total : 228426 time to create 1 rle with new method : 0.7259409427642822 time for calcul the mask position with numpy : 0.04041862487792969 nb_pixel_total : 50112 time to create 1 rle with old method : 0.05289435386657715 time for calcul the mask position with numpy : 0.04812955856323242 nb_pixel_total : 170201 time to create 1 rle with new method : 0.6318526268005371 time for calcul the mask position with numpy : 0.04108071327209473 nb_pixel_total : 23365 time to create 1 rle with old method : 0.02551412582397461 create new chi : 4.422037124633789 time to delete rle : 0.0014383792877197266 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 10783 TO DO : save crop sub photo not yet done ! save time : 0.6049244403839111 nb_obj : 12 nb_hashtags : 6 time to prepare the origin masks : 12.322496175765991 time for calcul the mask position with numpy : 0.4834880828857422 nb_pixel_total : 5955098 time to create 1 rle with new method : 0.6794705390930176 time for calcul the mask position with numpy : 0.04386401176452637 nb_pixel_total : 108537 time to create 1 rle with old method : 0.14816808700561523 time for calcul the mask position with numpy : 0.053545236587524414 nb_pixel_total : 153774 time to create 1 rle with new method : 0.42726612091064453 time for calcul the mask position with numpy : 0.044104576110839844 nb_pixel_total : 11084 time to create 1 rle with old method : 0.013253450393676758 time for calcul the mask position with numpy : 0.04292607307434082 nb_pixel_total : 202992 time to create 1 rle with new method : 0.6377620697021484 time for calcul the mask position with numpy : 0.0441131591796875 nb_pixel_total : 150366 time to create 1 rle with new method : 0.6371967792510986 time for calcul the mask position with numpy : 0.044506072998046875 nb_pixel_total : 57397 time to create 1 rle with old method : 0.06966328620910645 time for calcul the mask position with numpy : 0.04719686508178711 nb_pixel_total : 292286 time to create 1 rle with new method : 1.0467331409454346 time for calcul the mask position with numpy : 0.04519772529602051 nb_pixel_total : 15062 time to create 1 rle with old method : 0.017863750457763672 time for calcul the mask position with numpy : 0.044467926025390625 nb_pixel_total : 155939 time to create 1 rle with new method : 0.925128698348999 time for calcul the mask position with numpy : 0.04037141799926758 nb_pixel_total : 41729 time to create 1 rle with old method : 0.046666622161865234 time for calcul the mask position with numpy : 0.049776315689086914 nb_pixel_total : 772965 time to create 1 rle with new method : 0.6137580871582031 time for calcul the mask position with numpy : 0.04346060752868652 nb_pixel_total : 377171 time to create 1 rle with new method : 1.2717163562774658 create new chi : 7.810927391052246 time to delete rle : 0.0024518966674804688 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 18985 TO DO : save crop sub photo not yet done ! save time : 1.0504150390625 nb_obj : 13 nb_hashtags : 6 time to prepare the origin masks : 16.679638147354126 time for calcul the mask position with numpy : 0.7079248428344727 nb_pixel_total : 6114230 time to create 1 rle with new method : 0.8537309169769287 time for calcul the mask position with numpy : 0.041730403900146484 nb_pixel_total : 143518 time to create 1 rle with old method : 0.18976712226867676 time for calcul the mask position with numpy : 0.054256439208984375 nb_pixel_total : 25123 time to create 1 rle with old method : 0.028161048889160156 time for calcul the mask position with numpy : 0.04796266555786133 nb_pixel_total : 361394 time to create 1 rle with new method : 0.6587626934051514 time for calcul the mask position with numpy : 0.043039560317993164 nb_pixel_total : 420888 time to create 1 rle with new method : 0.890549898147583 time for calcul the mask position with numpy : 0.04442548751831055 nb_pixel_total : 81758 time to create 1 rle with old method : 0.09206128120422363 time for calcul the mask position with numpy : 0.04584956169128418 nb_pixel_total : 486452 time to create 1 rle with new method : 1.8977327346801758 time for calcul the mask position with numpy : 0.04174685478210449 nb_pixel_total : 9993 time to create 1 rle with old method : 0.011037111282348633 time for calcul the mask position with numpy : 0.027854204177856445 nb_pixel_total : 18054 time to create 1 rle with old method : 0.019740819931030273 time for calcul the mask position with numpy : 0.026908159255981445 nb_pixel_total : 110497 time to create 1 rle with old method : 0.12414002418518066 time for calcul the mask position with numpy : 0.030362606048583984 nb_pixel_total : 115512 time to create 1 rle with old method : 0.15175151824951172 time for calcul the mask position with numpy : 0.027876615524291992 nb_pixel_total : 33246 time to create 1 rle with old method : 0.041376352310180664 time for calcul the mask position with numpy : 0.02844715118408203 nb_pixel_total : 191102 time to create 1 rle with new method : 1.8347125053405762 time for calcul the mask position with numpy : 0.046054840087890625 nb_pixel_total : 182633 time to create 1 rle with new method : 1.529529333114624 create new chi : 9.74464726448059 time to delete rle : 0.0015490055084228516 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 16258 TO DO : save crop sub photo not yet done ! save time : 0.9205360412597656 nb_obj : 7 nb_hashtags : 5 time to prepare the origin masks : 10.918030023574829 time for calcul the mask position with numpy : 0.4869849681854248 nb_pixel_total : 7064484 time to create 1 rle with new method : 0.6405787467956543 time for calcul the mask position with numpy : 0.04426908493041992 nb_pixel_total : 257333 time to create 1 rle with new method : 0.634479284286499 time for calcul the mask position with numpy : 0.043891191482543945 nb_pixel_total : 9167 time to create 1 rle with old method : 0.01058506965637207 time for calcul the mask position with numpy : 0.04195237159729004 nb_pixel_total : 242969 time to create 1 rle with new method : 0.5306508541107178 time for calcul the mask position with numpy : 0.04508638381958008 nb_pixel_total : 416383 time to create 1 rle with new method : 0.9008986949920654 time for calcul the mask position with numpy : 0.04666399955749512 nb_pixel_total : 59578 time to create 1 rle with old method : 0.06859779357910156 time for calcul the mask position with numpy : 0.030810117721557617 nb_pixel_total : 229357 time to create 1 rle with new method : 0.651242733001709 time for calcul the mask position with numpy : 0.05314135551452637 nb_pixel_total : 15129 time to create 1 rle with old method : 0.02341294288635254 create new chi : 4.430049896240234 time to delete rle : 0.0014200210571289062 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 9772 TO DO : save crop sub photo not yet done ! save time : 0.5546050071716309 nb_obj : 15 nb_hashtags : 5 time to prepare the origin masks : 12.930416584014893 time for calcul the mask position with numpy : 0.25127196311950684 nb_pixel_total : 5455593 time to create 1 rle with new method : 0.40915942192077637 time for calcul the mask position with numpy : 0.03100752830505371 nb_pixel_total : 10244 time to create 1 rle with old method : 0.013566017150878906 time for calcul the mask position with numpy : 0.034731388092041016 nb_pixel_total : 353686 time to create 1 rle with new method : 0.42423081398010254 time for calcul the mask position with numpy : 0.026412487030029297 nb_pixel_total : 22895 time to create 1 rle with old method : 0.02685856819152832 time for calcul the mask position with numpy : 0.027074813842773438 nb_pixel_total : 49742 time to create 1 rle with old method : 0.06380224227905273 time for calcul the mask position with numpy : 0.036859750747680664 nb_pixel_total : 698688 time to create 1 rle with new method : 0.45318055152893066 time for calcul the mask position with numpy : 0.06018805503845215 nb_pixel_total : 338719 time to create 1 rle with new method : 0.8064734935760498 time for calcul the mask position with numpy : 0.04440617561340332 nb_pixel_total : 9904 time to create 1 rle with old method : 0.012571334838867188 time for calcul the mask position with numpy : 0.04129743576049805 nb_pixel_total : 2816 time to create 1 rle with old method : 0.003584623336791992 time for calcul the mask position with numpy : 0.043612003326416016 nb_pixel_total : 115817 time to create 1 rle with old method : 0.1319880485534668 time for calcul the mask position with numpy : 0.04494452476501465 nb_pixel_total : 85475 time to create 1 rle with old method : 0.09107494354248047 time for calcul the mask position with numpy : 0.04280209541320801 nb_pixel_total : 501807 time to create 1 rle with new method : 0.4415316581726074 time for calcul the mask position with numpy : 0.04152631759643555 nb_pixel_total : 109965 time to create 1 rle with old method : 0.12307143211364746 time for calcul the mask position with numpy : 0.03518486022949219 nb_pixel_total : 163202 time to create 1 rle with new method : 0.5679452419281006 time for calcul the mask position with numpy : 0.029175996780395508 nb_pixel_total : 239748 time to create 1 rle with new method : 0.37897467613220215 time for calcul the mask position with numpy : 0.04128241539001465 nb_pixel_total : 136099 time to create 1 rle with old method : 0.1555774211883545 create new chi : 5.137662649154663 time to delete rle : 0.002013683319091797 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 17316 TO DO : save crop sub photo not yet done ! save time : 0.9803838729858398 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 11.415789604187012 time for calcul the mask position with numpy : 0.7172563076019287 nb_pixel_total : 6285207 time to create 1 rle with new method : 2.339287281036377 time for calcul the mask position with numpy : 0.029132843017578125 nb_pixel_total : 431625 time to create 1 rle with new method : 0.7555041313171387 time for calcul the mask position with numpy : 0.028338909149169922 nb_pixel_total : 142011 time to create 1 rle with old method : 0.15856194496154785 time for calcul the mask position with numpy : 0.02989935874938965 nb_pixel_total : 217669 time to create 1 rle with new method : 0.9573848247528076 time for calcul the mask position with numpy : 0.030257701873779297 nb_pixel_total : 75318 time to create 1 rle with old method : 0.08765959739685059 time for calcul the mask position with numpy : 0.02869272232055664 nb_pixel_total : 169738 time to create 1 rle with new method : 0.6530444622039795 time for calcul the mask position with numpy : 0.029130220413208008 nb_pixel_total : 427240 time to create 1 rle with new method : 0.792255163192749 time for calcul the mask position with numpy : 0.03040003776550293 nb_pixel_total : 53133 time to create 1 rle with old method : 0.06484007835388184 time for calcul the mask position with numpy : 0.03038954734802246 nb_pixel_total : 87859 time to create 1 rle with old method : 0.10009574890136719 time for calcul the mask position with numpy : 0.03191494941711426 nb_pixel_total : 404600 time to create 1 rle with new method : 1.5239648818969727 create new chi : 8.636899709701538 time to delete rle : 0.0013132095336914062 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 13391 TO DO : save crop sub photo not yet done ! save time : 0.7425909042358398 map_output_result : {1327679562: (0.0, 'Should be the crop_list due to order', 0), 1327679550: (0.0, 'Should be the crop_list due to order', 0), 1327679546: (0.0, 'Should be the crop_list due to order', 0), 1327679541: (0.0, 'Should be the crop_list due to order', 0), 1327679536: (0.0, 'Should be the crop_list due to order', 0), 1327679521: (0.0, 'Should be the crop_list due to order', 0), 1327679518: (0.0, 'Should be the crop_list due to order', 0), 1327679517: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1327679562, 1327679550, 1327679546, 1327679541, 1327679536, 1327679521, 1327679518, 1327679517] Looping around the photos to save general results len do output : 8 /1327679562.Didn't retrieve data . /1327679550.Didn't retrieve data . /1327679546.Didn't retrieve data . /1327679541.Didn't retrieve data . /1327679536.Didn't retrieve data . /1327679521.Didn't retrieve data . /1327679518.Didn't retrieve data . /1327679517.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679562', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679550', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679546', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679541', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679536', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679521', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679518', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679517', None, None, None, None, None, '2647393') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 24 time used for this insertion : 0.014117956161499023 save_final save missing photos in datou_result : time spend for datou_step_exec : 143.17287349700928 time spend to save output : 0.10376262664794922 total time spend for step 5 : 143.27663612365723 step6:crop_condition Fri Mar 7 08:57:37 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 8 ! batch 1 Loaded 80 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741334261_1754908 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.4367766380310059 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! About to upload 10 photos upload in portfolio : 4869462 init cache_photo without model_param we have 10 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741334274_1754908 we have uploaded 10 photos in the portfolio 4869462 time of upload the photos Elapsed time : 22.984580755233765 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741334302_1754908 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 12.335658550262451 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741334317_1754908 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9466125965118408 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! About to upload 11 photos upload in portfolio : 4869462 init cache_photo without model_param we have 11 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741334332_1754908 we have uploaded 11 photos in the portfolio 4869462 time of upload the photos Elapsed time : 5.217020273208618 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1327679562, 1327679550, 1327679546, 1327679541, 1327679536, 1327679521, 1327679518, 1327679517] Looping around the photos to save general results len do output : 30 /1341417150Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417151Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417153Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417154Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417170Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417173Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417174Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417175Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417176Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417177Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417178Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417179Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417180Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417181Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417184Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417185Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417186Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417189Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417191Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417219Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417220Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417221Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417222Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417223Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417224Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417225Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417226Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417227Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417228Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341417229Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679562', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679550', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679546', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679541', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679536', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679521', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679518', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679517', None, None, None, None, None, '2647393') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 98 time used for this insertion : 0.17288732528686523 save_final save missing photos in datou_result : time spend for datou_step_exec : 85.4550838470459 time spend to save output : 0.2562391757965088 total time spend for step 6 : 85.71132302284241 step7:ventilate_hashtags_in_portfolio Fri Mar 7 08:59:03 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 21136611 get user id for portfolio 21136611 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21136611 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','autre_refus','environnement','Carton_gris','cartonnette','mal_croppe','plastique','Teint_Dans_La_Masse','papier','kraft','Carton_brun','metal')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21136611 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','autre_refus','environnement','Carton_gris','cartonnette','mal_croppe','plastique','Teint_Dans_La_Masse','papier','kraft','Carton_brun','metal')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21136611 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','autre_refus','environnement','Carton_gris','cartonnette','mal_croppe','plastique','Teint_Dans_La_Masse','papier','kraft','Carton_brun','metal')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/21137012,21137013,21137014,21137015,21137016,21137017,21137018,21137019,21137020,21137021,21137022,21137023?tags=flou,autre_refus,environnement,Carton_gris,cartonnette,mal_croppe,plastique,Teint_Dans_La_Masse,papier,kraft,Carton_brun,metal Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1327679562, 1327679550, 1327679546, 1327679541, 1327679536, 1327679521, 1327679518, 1327679517] Looping around the photos to save general results len do output : 1 /21136611. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679562', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679550', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679546', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679541', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679536', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679521', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679518', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679517', None, None, None, None, None, '2647393') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 9 time used for this insertion : 0.013538837432861328 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.4852194786071777 time spend to save output : 0.01393437385559082 total time spend for step 7 : 3.4991538524627686 step8:final Fri Mar 7 08:59:06 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1327679562: ('0.1896752411517101',), 1327679550: ('0.1896752411517101',), 1327679546: ('0.1896752411517101',), 1327679541: ('0.1896752411517101',), 1327679536: ('0.1896752411517101',), 1327679521: ('0.1896752411517101',), 1327679518: ('0.1896752411517101',), 1327679517: ('0.1896752411517101',)} new output for save of step final : {1327679562: ('0.1896752411517101',), 1327679550: ('0.1896752411517101',), 1327679546: ('0.1896752411517101',), 1327679541: ('0.1896752411517101',), 1327679536: ('0.1896752411517101',), 1327679521: ('0.1896752411517101',), 1327679518: ('0.1896752411517101',), 1327679517: ('0.1896752411517101',)} [1327679562, 1327679550, 1327679546, 1327679541, 1327679536, 1327679521, 1327679518, 1327679517] Looping around the photos to save general results len do output : 8 /1327679562.Didn't retrieve data . /1327679550.Didn't retrieve data . /1327679546.Didn't retrieve data . /1327679541.Didn't retrieve data . /1327679536.Didn't retrieve data . /1327679521.Didn't retrieve data . /1327679518.Didn't retrieve data . /1327679517.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679562', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679550', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679546', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679541', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679536', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679521', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679518', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679517', None, None, None, None, None, '2647393') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 24 time used for this insertion : 0.02656269073486328 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10994791984558105 time spend to save output : 0.027088165283203125 total time spend for step 8 : 0.13703608512878418 step9:velours_tree Fri Mar 7 08:59:06 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.1969757080078125 time spend to save output : 4.3392181396484375e-05 total time spend for step 9 : 0.19701910018920898 step10:send_mail_cod Fri Mar 7 08:59:06 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21136611_07-03-2025_08_59_07.pdf 21137012 imagette211370121741334347 21137013 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211370131741334347 21137016 imagette211370161741334347 21137017 imagette211370171741334348 21137018 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211370181741334348 21137021 imagette211370211741334349 21137023 imagette211370231741334349 21137015 change filename to text .change filename to text .change filename to text .imagette211370151741334349 21137019 change filename to text .change filename to text .change filename to text .change filename to text .imagette211370191741334349 21137022 change filename to text .change filename to text .imagette211370221741334349 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21136611 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21137012,21137013,21137014,21137015,21137016,21137017,21137018,21137019,21137020,21137021,21137022,21137023?tags=flou,autre_refus,environnement,Carton_gris,cartonnette,mal_croppe,plastique,Teint_Dans_La_Masse,papier,kraft,Carton_brun,metal your option no_mail is active, we will not send the real mail to your client args[1327679562] : ('0.1896752411517101',) We are sending mail with results at report@fotonower.com args[1327679550] : ('0.1896752411517101',) We are sending mail with results at report@fotonower.com args[1327679546] : ('0.1896752411517101',) We are sending mail with results at report@fotonower.com args[1327679541] : ('0.1896752411517101',) We are sending mail with results at report@fotonower.com args[1327679536] : ('0.1896752411517101',) We are sending mail with results at report@fotonower.com args[1327679521] : ('0.1896752411517101',) We are sending mail with results at report@fotonower.com args[1327679518] : ('0.1896752411517101',) We are sending mail with results at report@fotonower.com args[1327679517] : ('0.1896752411517101',) We are sending mail with results at report@fotonower.com refus_total : 0.1896752411517101 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=21136611 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21136611_07-03-2025_08_59_07.pdf results_Auto_P21136611_07-03-2025_08_59_07.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21136611_07-03-2025_08_59_07.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','21136611','results_Auto_P21136611_07-03-2025_08_59_07.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21136611_07-03-2025_08_59_07.pdf','pdf','','1.05','0.1896752411517101') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1327679562, 1327679550, 1327679546, 1327679541, 1327679536, 1327679521, 1327679518, 1327679517] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679562', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679550', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679546', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679541', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679536', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679521', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679518', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679517', None, None, None, None, None, '2647393') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 8 time used for this insertion : 0.01380014419555664 save_final save missing photos in datou_result : time spend for datou_step_exec : 5.474886894226074 time spend to save output : 0.014070510864257812 total time spend for step 10 : 5.488957405090332 step11:split_time_score Fri Mar 7 08:59:12 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('17', 8),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 09012025 21136611 Nombre de photos uploadées : 8 / 23040 (0%) 09012025 21136611 Nombre de photos taguées (types de déchets): 0 / 8 (0%) 09012025 21136611 Nombre de photos taguées (volume) : 0 / 8 (0%) elapsed_time : load_data_split_time_score 3.337860107421875e-06 elapsed_time : order_list_meta_photo_and_scores 8.106231689453125e-06 ???????? elapsed_time : fill_and_build_computed_from_old_data 0.0005316734313964844 elapsed_time : insert_dashboard_record_day_entry 0.02227473258972168 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.035256224334702845 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21136549_07-03-2025_08_49_18.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136549 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21136549 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136550 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136551 order by id desc limit 1 Qualite : 0.0325748641699712 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19485837_09-01-2025_09_15_00.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19485837 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19485837 AND mptpi.`type`=4200 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136552 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136553 order by id desc limit 1 Qualite : 0.025699729488632497 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19487949_10-01-2025_20_23_59.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19487949 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11512 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11521 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11520 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11516 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11516 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11519 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11519 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11513 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11513 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11577 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11577 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11515 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11514 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11514 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11523 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11515 doesn't seem to be define in the database( WARNING : type of input 3 of step 11514 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11512 doesn't seem to be define in the database( WARNING : type of input 2 of step 11516 doesn't seem to be define in the database( WARNING : output 1 of step 11512 have datatype=2 whereas input 1 of step 11519 have datatype=7 WARNING : type of output 2 of step 11519 doesn't seem to be define in the database( WARNING : type of input 1 of step 11513 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11515 have datatype=10 whereas input 3 of step 11522 have datatype=6 WARNING : type of input 2 of step 11577 doesn't seem to be define in the database( WARNING : output 1 of step 11513 have datatype=7 whereas input 2 of step 11577 have datatype=None WARNING : type of output 3 of step 11577 doesn't seem to be define in the database( WARNING : type of input 1 of step 11515 doesn't seem to be define in the database( WARNING : output 0 of step 11515 have datatype=10 whereas input 0 of step 11583 have datatype=18 WARNING : type of input 5 of step 11522 doesn't seem to be define in the database( WARNING : output 0 of step 11583 have datatype=11 whereas input 5 of step 11522 have datatype=None WARNING : type of output 1 of step 11521 doesn't seem to be define in the database( WARNING : type of input 3 of step 11516 doesn't seem to be define in the database( WARNING : type of output 1 of step 11520 doesn't seem to be define in the database( WARNING : type of input 3 of step 11516 doesn't seem to be define in the database( WARNING : output 0 of step 11519 have datatype=1 whereas input 0 of step 11513 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19487949 AND mptpi.`type`=4203 To do Qualite : 0.003272806685072594 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19487950_09-01-2025_11_38_19.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19487950 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19487950 AND mptpi.`type`=3327 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136554 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136555 order by id desc limit 1 Qualite : 0.0028876571375018767 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19487954_10-01-2025_21_06_24.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19487954 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11524 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11533 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11532 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11528 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11528 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11531 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11531 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11578 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11578 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11527 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11526 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11526 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11535 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11527 doesn't seem to be define in the database( WARNING : type of input 3 of step 11526 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11524 doesn't seem to be define in the database( WARNING : type of input 2 of step 11528 doesn't seem to be define in the database( WARNING : output 1 of step 11524 have datatype=2 whereas input 1 of step 11531 have datatype=7 WARNING : type of output 2 of step 11531 doesn't seem to be define in the database( WARNING : type of input 1 of step 11525 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11527 have datatype=10 whereas input 3 of step 11534 have datatype=6 WARNING : type of input 2 of step 11578 doesn't seem to be define in the database( WARNING : output 1 of step 11525 have datatype=7 whereas input 2 of step 11578 have datatype=None WARNING : type of output 3 of step 11578 doesn't seem to be define in the database( WARNING : type of input 1 of step 11527 doesn't seem to be define in the database( WARNING : output 0 of step 11527 have datatype=10 whereas input 0 of step 11584 have datatype=18 WARNING : type of input 5 of step 11534 doesn't seem to be define in the database( WARNING : output 0 of step 11584 have datatype=11 whereas input 5 of step 11534 have datatype=None WARNING : type of output 1 of step 11533 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : type of output 1 of step 11532 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : output 0 of step 11531 have datatype=1 whereas input 0 of step 11525 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19487954 AND mptpi.`type`=4211 To do Qualite : 0.05091045257650593 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19487955_09-01-2025_11_05_08.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19487955 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19487955 AND mptpi.`type`=4200 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136556 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136557 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136558 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136559 order by id desc limit 1 Qualite : 0.005372028363486408 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19489650_09-01-2025_20_49_36.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19489650 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19489650 AND mptpi.`type`=3327 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136560 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136561 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136562 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136563 order by id desc limit 1 Qualite : 0.0023407997443465633 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19492194_10-01-2025_23_23_28.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19492194 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11524 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11533 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11532 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11528 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11528 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11531 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11531 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11578 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11578 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11527 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11526 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11526 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11535 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11527 doesn't seem to be define in the database( WARNING : type of input 3 of step 11526 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11524 doesn't seem to be define in the database( WARNING : type of input 2 of step 11528 doesn't seem to be define in the database( WARNING : output 1 of step 11524 have datatype=2 whereas input 1 of step 11531 have datatype=7 WARNING : type of output 2 of step 11531 doesn't seem to be define in the database( WARNING : type of input 1 of step 11525 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11527 have datatype=10 whereas input 3 of step 11534 have datatype=6 WARNING : type of input 2 of step 11578 doesn't seem to be define in the database( WARNING : output 1 of step 11525 have datatype=7 whereas input 2 of step 11578 have datatype=None WARNING : type of output 3 of step 11578 doesn't seem to be define in the database( WARNING : type of input 1 of step 11527 doesn't seem to be define in the database( WARNING : output 0 of step 11527 have datatype=10 whereas input 0 of step 11584 have datatype=18 WARNING : type of input 5 of step 11534 doesn't seem to be define in the database( WARNING : output 0 of step 11584 have datatype=11 whereas input 5 of step 11534 have datatype=None WARNING : type of output 1 of step 11533 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : type of output 1 of step 11532 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : output 0 of step 11531 have datatype=1 whereas input 0 of step 11525 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19492194 AND mptpi.`type`=4211 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136564 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136565 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136566 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136567 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136568 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136569 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136570 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136571 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136572 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136573 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136574 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136575 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136576 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136577 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136578 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136580 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136581 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136582 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136583 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136584 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136585 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136586 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136587 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136588 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136589 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136590 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136592 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136593 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136594 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136595 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136596 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136598 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136599 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136600 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136601 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136602 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136603 order by id desc limit 1 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21136604_07-03-2025_08_59_18.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136604 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136605 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136606 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136607 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136608 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136609 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136610 order by id desc limit 1 Qualite : 0.1896752411517101 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21136611_07-03-2025_08_59_07.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136611 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21136611 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136612 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136613 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136614 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136615 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136616 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136617 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136618 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136619 order by id desc limit 1 Qualite : 0.18365381514099943 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21136620_07-03-2025_08_53_40.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136620 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21136620 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136621 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136622 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136623 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136624 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136625 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136626 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136627 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136628 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136629 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136630 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136631 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136632 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136633 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136634 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136635 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136636 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136637 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136638 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136639 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136640 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136641 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21136642 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'09012025': {'nb_upload': 8, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1327679562, 1327679550, 1327679546, 1327679541, 1327679536, 1327679521, 1327679518, 1327679517] Looping around the photos to save general results len do output : 1 /21136611Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679562', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679550', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679546', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679541', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679536', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679521', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679518', None, None, None, None, None, '2647393') ('3459', None, None, None, None, None, None, None, '2647393') ('3459', '21136611', '1327679517', None, None, None, None, None, '2647393') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 9 time used for this insertion : 0.012891054153442383 save_final save missing photos in datou_result : time spend for datou_step_exec : 72.75090980529785 time spend to save output : 0.013199329376220703 total time spend for step 11 : 72.76410913467407 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 1 len(input) + len(total_photo_id_missing) : 8 set_done_treatment 187.16user 166.96system 8:01.88elapsed 73%CPU (0avgtext+0avgdata 3752736maxresident)k 378840inputs+232416outputs (24052major+14021801minor)pagefaults 0swaps