python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 25333 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 3184>200 please execute split_portfolio.py -i 26380249 -l 200 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 533, datou_cur_ids : ['3643877'] with mtr_portfolio_ids : ['26380249'] and first list_photo_ids : [] new path : /proc/25333/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 95 ; length of list_pids : 95 ; length of list_args : 95 time to download the photos : 13.939252138137817 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Mon Sep 1 16:20:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2406 wait 20 seconds inside check gpu memory 2025-09-01 16:21:12.532089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-09-01 16:21:12.565723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-01 16:21:12.566246: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-01 16:21:12.569318: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-01 16:21:12.571560: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-01 16:21:12.571937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-01 16:21:12.574488: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-01 16:21:12.575614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-01 16:21:12.580423: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-01 16:21:12.581630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-01 16:21:12.582266: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-09-01 16:21:12.608575: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-09-01 16:21:12.610875: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8458000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-09-01 16:21:12.610935: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-09-01 16:21:12.747980: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1db9ca70 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-09-01 16:21:12.748040: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-09-01 16:21:12.748924: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-01 16:21:12.749052: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-01 16:21:12.749083: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-01 16:21:12.749106: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-01 16:21:12.749132: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-01 16:21:12.749157: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-01 16:21:12.749182: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-01 16:21:12.749206: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-01 16:21:12.750324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-01 16:21:12.750388: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-01 16:21:12.751211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-01 16:21:12.751233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-01 16:21:12.751245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-01 16:21:12.752575: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3545 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.917518854141235 time used to load_weights : 0.28203606605529785 0it [00:00, ?it/s] 64it [00:00, 631.09it/s] 95it [00:00, 634.49it/s]2025-09-01 16:21:24.460811: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-01 16:21:24.681343: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-01 16:21:24.991461: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-09-01 16:21:24.993869: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1756736429_25333_1380657041_4daf09f1ab92fd468e6f87fdc293d931_42560268.jpg temp/1756736429_25333_1380657040_4bf2da600c01bc4305d501a7e0603e74_65444421.jpg temp/1756736429_25333_1380657039_f540b3b60e0f2b6318aaa2691ac721ae_55956667.jpg temp/1756736429_25333_1380657038_143b7c5c458ab984c90284d0409df609_70930208.jpg temp/1756736429_25333_1380657037_6eb3f4b1f0a935e0b21c55c804e632c7_32425818.jpg temp/1756736429_25333_1380657036_6021c99099dde39e213448e8f4b964df_2910013.jpg temp/1756736429_25333_1380656973_a08b395af5dba30b72d44bd1491903ec_26622510.jpg temp/1756736429_25333_1380656972_a50a2a978908ce28bc381e6c9a3c20ea_27538228.jpg temp/1756736429_25333_1380656971_063990999a7525692825d1bf8ca75426_5257857.jpg temp/1756736429_25333_1380656970_e32a9e0853ba74722de771b06c7747b6_57632388.jpg temp/1756736429_25333_1380656968_f83f66b164c14024d7cd9d6f24ef0c98_3407353.jpg temp/1756736429_25333_1380656967_8396983b9b92cee01c2eb6412eea5537_78375366.jpg temp/1756736429_25333_1380656586_9d40e50dab89462de32903786089baa2_69474025.jpg temp/1756736429_25333_1380656585_d3c76d491b0176bb5d273b8980966bce_54247102.jpg temp/1756736429_25333_1380656584_3feb1ad46e06f7a7193370faf7575f15_34615713.jpg temp/1756736429_25333_1380656583_838ca7728839c8cee27665d12873b10e_82101339.jpg temp/1756736429_25333_1380656582_71c47e271627989bdd3cb14ef54cb83c_31358615.jpg temp/1756736429_25333_1380656581_5137aa1bdbf5721c0973f56777cb7327_90811946.jpg temp/1756736429_25333_1380643144_259a5dca59042b04873afa4b03aa7a17_78465827.jpg temp/1756736429_25333_1380643143_bca60002c140842ac055e4adbde8472c_90125470.jpg temp/1756736429_25333_1380643142_a1386b43679f7db5584b0f663cf99a89_94939075.jpg temp/1756736429_25333_1380643141_33dccdccd948653a66bce605a1722b27_51398856.jpg temp/1756736429_25333_1380643140_8eaecf24fa68a180d00866e4fef768e4_76104491.jpg temp/1756736429_25333_1380642684_b1dc54dfe7be89700669316cfb52da6e_38703967.jpg temp/1756736429_25333_1380642680_259a5dca59042b04873afa4b03aa7a17_28314133.jpg temp/1756736429_25333_1380642677_bca60002c140842ac055e4adbde8472c_98936518.jpg temp/1756736429_25333_1380641920_0e52ada06350fa2f8fd77b242a0c3d46_76926928.jpg temp/1756736429_25333_1380641918_f9b9146b57d796269a36196097f7c783_336826.jpg temp/1756736429_25333_1380641916_e18601bfc04fa00c8820430fa23428a5_85960011.jpg temp/1756736429_25333_1380641341_69fbdbca6adc27be239e5f9d6eaea493_22027824.jpg temp/1756736429_25333_1380641339_00f24b90da00946ddb474970bbd2b092_45079050.jpg temp/1756736429_25333_1380641337_3d04e21da288ac3e831802083b9f1027_10375388.jpg temp/1756736429_25333_1380640956_f2befc4437d08f30fdf90296c73c1668_25146385.jpg temp/1756736429_25333_1380640954_94a41721ae02ac4916af4276d3999124_30810859.jpg temp/1756736429_25333_1380640952_4ce38a3d775a7382dacb42071eac7c14_86683625.jpg temp/1756736429_25333_1380640428_3d540f03d55891cb83cb0ab688418a4b_8769170.jpg temp/1756736429_25333_1380640426_7537f875e40f43ccfeff22e480abfe0f_87827205.jpg temp/1756736429_25333_1380640424_7f0dafd790f799e73050cd5a474336d5_59757072.jpg temp/1756736429_25333_1380640370_da0384848f1e57db791006ce8435d3ba_79369528.jpg temp/1756736429_25333_1380640369_9498664882671c5cf33507e321f0657d_38478925.jpg temp/1756736429_25333_1380640367_6aa40dc4c7d20a358653ed60b8574a23_85093527.jpg temp/1756736429_25333_1380640327_a3ba1a3f3d0d55bc55fac43f477f635b_62773665.jpg temp/1756736429_25333_1380640326_4a1bfb1197003ecc9a560030496b2697_85047603.jpg temp/1756736429_25333_1380640325_1527f9bb3f48b80647394ddc5d9fa0d5_86259237.jpg temp/1756736429_25333_1380640286_56d0e2109769b38c7b9461576bb6a0ca_55398221.jpg temp/1756736429_25333_1380640284_73c4c0a201fd5c378aeec47345e3a4ab_67839924.jpg temp/1756736429_25333_1380640282_392ce09d56307feb54ac1928095a8e71_66636392.jpg temp/1756736429_25333_1380640145_4026039fec8aa38375074019cf63c384_93480504.jpg temp/1756736429_25333_1380640143_69595eab8c0249c670f9617a5af3bd2f_14819174.jpg temp/1756736429_25333_1380640140_91136da82573bcd2104c6418f4b68e0a_58683498.jpg temp/1756736429_25333_1380640112_b464e4750d6ddcee08d9f7cd1a3881e4_55188672.jpg temp/1756736429_25333_1380640111_929aba4fb2000627fd6e03a004d5aae1_22873075.jpg temp/1756736429_25333_1380640110_808eb1f315d53c9c4508413ed4c01d89_85006064.jpg temp/1756736429_25333_1380640109_25ebe0c868268f4844929c8dc455bf09_78277238.jpg temp/1756736429_25333_1380640108_c5979cbb4650e6ab936eed7c64ac4a8f_26037934.jpg temp/1756736429_25333_1380640107_e5653f94fc2fd0def7ed6c8797209f0c_27315591.jpg temp/1756736429_25333_1380639822_b464e4750d6ddcee08d9f7cd1a3881e4_37456594.jpg temp/1756736429_25333_1380639821_929aba4fb2000627fd6e03a004d5aae1_96500366.jpg temp/1756736429_25333_1380639820_808eb1f315d53c9c4508413ed4c01d89_81117845.jpg temp/1756736429_25333_1380639789_64e28c146250617eb90d1e24e2aa3282_83566235.jpg temp/1756736429_25333_1380639788_7ac40b54bff12aea950be5b7d64fafe1_87156696.jpg temp/1756736429_25333_1380639787_1518cb71415fae05d937eb6b6558c161_92955948.jpg temp/1756736429_25333_1380639711_0674f32256a3364492c2c06c723ad421_68006432.jpg temp/1756736429_25333_1380639710_d29db3cf1b1c5dbd2316f3dd016495f5_78006594.jpg temp/1756736429_25333_1380639709_da7d5311533bea46f9cc3e5705914f9c_90965426.jpg temp/1756736429_25333_1380639669_1f8c08c377d424fe51ff28ad097f8a1a_45851613.jpg temp/1756736429_25333_1380639668_3454c32b68a661bdbaf3b7402d066749_17075313.jpg temp/1756736429_25333_1380639667_930bbe9d1c9ccd7557b298b9fa65134b_37343771.jpg temp/1756736429_25333_1380639607_04b26f1f9e7866d91d15e47bcadd4355_86546647.jpg temp/1756736429_25333_1380639606_a98243e2d1b5b7d13901e19ab078f93e_71669966.jpg temp/1756736429_25333_1380639605_687d2878325ef454b80c59c6545bca0c_24188913.jpg temp/1756736429_25333_1380639569_ae4b133d4b7b482f4d02084b45e5c477_75049946.jpg temp/1756736429_25333_1380639567_93cb038345afd56b44b539c302bd4b76_77381516.jpg temp/1756736429_25333_1380639566_5bde9e7cec5746eff65c057aab6f8b85_76121180.jpg temp/1756736429_25333_1380639502_e369df6f3834dfd992a649d0171a4ff5_70890409.jpg temp/1756736429_25333_1380639501_5d68574b2452e9048ff0f7cf7f5d65ba_16691442.jpg temp/1756736429_25333_1380639500_a2573e84ce109412121dd54b7cd2e9e1_42956614.jpg temp/1756736429_25333_1380639291_3111d46895786861c2d3b0e62beb32a0_11074290.jpg temp/1756736429_25333_1380639289_bab794f29e3c4f5ea8e059c1b482fed1_32000924.jpg temp/1756736429_25333_1380639286_75759f7d608f6365d5f0ba31b089712a_13919696.jpg temp/1756736429_25333_1380639099_4347fde3726c62d8ce4596286af3f8a8_21965855.jpg temp/1756736429_25333_1380639098_4bc75bc66e77ac6c3242d039c840756d_5355695.jpg temp/1756736429_25333_1380639096_d0957eeb6c4cce6c08f56bffba3cec3f_58153282.jpg temp/1756736429_25333_1380639014_1f46a2228c65e2e4ee0829c13f8c0aee_16925842.jpg temp/1756736429_25333_1380639013_7b211d782b7aaf1f44f945ea3e934fba_42918999.jpg temp/1756736429_25333_1380639012_0dd2cb9392ce55a4905748f93472c9e9_53066559.jpg temp/1756736429_25333_1380638879_9725d9a05f43450ceff8649a0dede5c4_11521731.jpg temp/1756736429_25333_1380638877_8e7d2dedbac1a9f240e7d4914f6d73a6_72926459.jpg temp/1756736429_25333_1380638874_d50302921d05e718ef96e1ff4414282b_58368082.jpg temp/1756736429_25333_1380638717_45809d06e4e994018eb0bcc7dd5f6dbf_60015269.jpg temp/1756736429_25333_1380638716_29d63e926402934953365ec8924d823d_4775139.jpg temp/1756736429_25333_1380638714_83184e91b42d6ca0de64005cca90e20d_14126132.jpg temp/1756736429_25333_1380638676_b6ab7371484f921735e52cbc08065581_32769510.jpg temp/1756736429_25333_1380638674_6898e48c3a50d49e8a19ae3b0984bfaf_13271579.jpg temp/1756736429_25333_1380638672_e4f7b302640ada4315e8c2e489dd3e58_57041620.jpg Found 95 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1380657041, 1380657040, 1380657039, 1380657038, 1380657037, 1380657036, 1380656973, 1380656972, 1380656971, 1380656970, 1380656968, 1380656967, 1380656586, 1380656585, 1380656584, 1380656583, 1380656582, 1380656581, 1380643144, 1380643143, 1380643142, 1380643141, 1380643140, 1380642684, 1380642680, 1380642677, 1380641920, 1380641918, 1380641916, 1380641341, 1380641339, 1380641337, 1380640956, 1380640954, 1380640952, 1380640428, 1380640426, 1380640424, 1380640370, 1380640369, 1380640367, 1380640327, 1380640326, 1380640325, 1380640286, 1380640284, 1380640282, 1380640145, 1380640143, 1380640140, 1380640112, 1380640111, 1380640110, 1380640109, 1380640108, 1380640107, 1380639822, 1380639821, 1380639820, 1380639789, 1380639788, 1380639787, 1380639711, 1380639710, 1380639709, 1380639669, 1380639668, 1380639667, 1380639607, 1380639606, 1380639605, 1380639569, 1380639567, 1380639566, 1380639502, 1380639501, 1380639500, 1380639291, 1380639289, 1380639286, 1380639099, 1380639098, 1380639096, 1380639014, 1380639013, 1380639012, 1380638879, 1380638877, 1380638874, 1380638717, 1380638716, 1380638714, 1380638676, 1380638674, 1380638672] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 95 time used for this insertion : 0.05602598190307617 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 41.34040641784668 time spend to save output : 0.06272006034851074 total time spend for step 0 : 41.40312647819519 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 15.75user 5.28system 1:02.46elapsed 33%CPU (0avgtext+0avgdata 2075868maxresident)k 581976inputs+125056outputs (273major+676485minor)pagefaults 0swaps