HTTP Error 503: Service Unavailable can't download the photo : 1348700528 complete_path : /home/admin/workarea/git/Velours/python/temp/1743291990_956648_1348700528_58fa78b232560b621e7fb8906483202d.jpg FBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBHTTP Error 503: Service Unavailable can't download the photo : 1348694301 complete_path : /home/admin/workarea/git/Velours/python/temp/1743291990_956648_1348694301_a1862e4ccf1a43723fed5f0d8fb6095e.jpg FBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 2 photos in the step downloads : photo missing : [1348700528, 1348694301] try to delete the photos missing in DB Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1348700528 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1348694301 length of list_filenames : 98 ; length of list_pids : 98 ; length of list_args : 98 time to download the photos : 135.47860097885132 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Sun Mar 30 00:48:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-03-30 00:48:47.313095: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-30 00:48:47.343246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-30 00:48:47.343516: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-30 00:48:47.346373: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-30 00:48:47.350232: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-30 00:48:47.350584: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-30 00:48:47.355741: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-30 00:48:47.358171: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-30 00:48:47.369304: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-30 00:48:47.370809: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-30 00:48:47.372405: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-30 00:48:47.403374: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-30 00:48:47.405507: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f1558000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-30 00:48:47.405578: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-30 00:48:47.620697: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x301a51f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-30 00:48:47.620745: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-30 00:48:47.621926: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-30 00:48:47.622026: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-30 00:48:47.622055: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-30 00:48:47.622083: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-30 00:48:47.622111: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-30 00:48:47.622138: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-30 00:48:47.622165: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-30 00:48:47.622192: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-30 00:48:47.623668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-30 00:48:47.623735: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-30 00:48:47.624658: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-30 00:48:47.624682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-30 00:48:47.624697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-30 00:48:47.626241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3516 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.511677265167236 time used to load_weights : 0.23247647285461426 0it [00:00, ?it/s] 98it [00:00, 1932.67it/s]2025-03-30 00:48:57.398961: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-30 00:48:57.603287: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-30 00:48:57.763347: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-03-30 00:48:57.778854: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1743291990_956648_1348703354_76ae670ff044b7581211c7bd2fe43baa.jpg temp/1743291990_956648_1348703328_d2c87f2d361c284a7435d6b1927d325f.jpg temp/1743291990_956648_1348703311_80643698f392191e4109654ef15672c1.jpg temp/1743291990_956648_1348703309_6912bce7451eb3ab31487f45718bba09.jpg temp/1743291990_956648_1348703306_c964f49fa40b1504c48eb3d5d5a90c64.jpg temp/1743291990_956648_1348703304_76d6706ad276a56e78d618e2bec3d57d.jpg temp/1743291990_956648_1348703301_21881d725f36b8ec51ef5e342e4ddfe8.jpg temp/1743291990_956648_1348703299_e506bf5a56e22517c7f5cc504f4fde99.jpg temp/1743291990_956648_1348703295_e854b1f037db9e93cbd9f1023bc43572.jpg temp/1743291990_956648_1348703267_73f7d23e0e1244a660164f9afb08a929.jpg temp/1743291990_956648_1348703045_2a508abc895d7ca0b0b3379fc4d49d34.jpg temp/1743291990_956648_1348702708_eb15cd063e29aad9ec7d59ae100c73ea.jpg temp/1743291990_956648_1348702152_5c5ccff8f5f092eaa85e9c73902d5990.jpg temp/1743291990_956648_1348702127_f90af8360d2045adf3d12543fb0386dd.jpg temp/1743291990_956648_1348701908_d91e7607a82fe8b6224aff7688fdfb2d.jpg temp/1743291990_956648_1348701906_8e30e4959a2b4ed4c0300d794f7df009.jpg temp/1743291990_956648_1348701905_4d5dc9a340d4ecd52aacc7f2bb84d6f6.jpg temp/1743291990_956648_1348701896_2f76177e62d5081be414e311e09d9ad7.jpg temp/1743291990_956648_1348701801_6dec7e44cd8d35e9a06a8debbacbfabc.jpg temp/1743291990_956648_1348701763_c51c4545cdb76a24a5b0530132d653ea.jpg temp/1743291990_956648_1348701718_ca3a322a8fc044e6d50d83c914af2d59.jpg temp/1743291990_956648_1348701669_0350bb050a63ce916c4b6af6f39c3889.jpg temp/1743291990_956648_1348701616_3090e0beab1a22d088fc2a31b805e2dc.jpg temp/1743291990_956648_1348701574_a2e3408a69e69db33c0ce15855b972a3.jpg temp/1743291990_956648_1348701505_625a280fb8d1a2c056a1af6e40da8946.jpg temp/1743291990_956648_1348701497_0e521ba1b6b0e76b5ee8323ec0b30815.jpg temp/1743291990_956648_1348701414_49149ef27f9eac7886e2696e2d7fed5c.jpg temp/1743291990_956648_1348701386_22f5207b2a529816b92579c16944c417.jpg temp/1743291990_956648_1348701346_459271043105dfd738f0937394d5795b.jpg temp/1743291990_956648_1348701286_ef213fd016fc71cecc5369b410730919.jpg temp/1743291990_956648_1348700948_64f64e69d431c16494f7d42a466f942b.jpg temp/1743291990_956648_1348700871_0ec63c7d0dc163c69cf94510235d309e.jpg temp/1743291990_956648_1348700838_00f61b1e61d33644a1b728f291406144.jpg temp/1743291990_956648_1348700811_227e623ca5ed662a2f1c7c6cc89923c5.jpg temp/1743291990_956648_1348700784_3b00ca2326e8ed84feb17d783448dffd.jpg temp/1743291990_956648_1348700742_ac00d0041a915bc6f790f2d9e2cbafb5.jpg temp/1743291990_956648_1348700492_02486ab289a1ef43cb1a9711bc6f8c96.jpg temp/1743291990_956648_1348700460_e6d674bf312178c0009a7d8822a428b5.jpg temp/1743291990_956648_1348700384_3af57be91e9309230b5ca205c9fd1492.jpg temp/1743291990_956648_1348700314_27ceecc51dfa32bb0ecb8e15cc89f696.jpg temp/1743291990_956648_1348700189_1c3e1c3cf4bbcfa64439d1ff2f7907b5.jpg temp/1743291990_956648_1348699349_fcb6677bd02a0819501d09316cd194ec.jpg temp/1743291990_956648_1348699241_91dd3c0fcfb2c16693bc3535d09459ee.jpg temp/1743291990_956648_1348698850_d8f7d908fff650fae3b74e2072ba42f3.jpg temp/1743291990_956648_1348698749_da5374804e06c2cbe228b4cecf6c24ae.jpg temp/1743291990_956648_1348698642_233d0e763f5a1717ead30d21ce902315.jpg temp/1743291990_956648_1348698576_f3fab3adacffe87448789c57c4b97c40.jpg temp/1743291990_956648_1348698362_970f0102576444fb1864477b91a6bfe5.jpg temp/1743291990_956648_1348698289_df6c3f96fcc4d70740844ae7af48a089.jpg temp/1743291990_956648_1348698196_6e911f6c7481d5b6c460c58a3d538537.jpg temp/1743291990_956648_1348698159_6a07e801243dfbb45a05345443c1018c.jpg temp/1743291990_956648_1348698101_348cfbaf78ae88f7c80146481528acc2.jpg temp/1743291990_956648_1348698097_9148b6c35bcd2c852e760eb89288e475.jpg temp/1743291990_956648_1348697851_e959661ca80b7a6bfe268593bf9c172b.jpg temp/1743291990_956648_1348697790_35c66a4851fabefd3a7802742892803c.jpg temp/1743291990_956648_1348697746_7d41196381b58d15f20d97c39e2a1bdf.jpg temp/1743291990_956648_1348697718_796d504cbeef1a5269dcebedf3b1fcb2.jpg temp/1743291990_956648_1348697679_bffe64a95f9bdbd7d9945df88d1003c2.jpg temp/1743291990_956648_1348697558_c4f6d96846c592b2b71a6bb70b26dc32.jpg temp/1743291990_956648_1348697463_9c1b4a38a4b5eb6a149ea280c6c4bdaa.jpg temp/1743291990_956648_1348697462_df4faaa0a2c6a40e585bfb71fd7abb38.jpg temp/1743291990_956648_1348697460_f9004e718dede9495d6e5d3eaa8dc954.jpg temp/1743291990_956648_1348697449_49f8aed59b3f65827f7bba972e2b52ee.jpg temp/1743291990_956648_1348697422_1722470efe562c1cd3535de24f0d91d1.jpg temp/1743291990_956648_1348696956_e3d07168f7310b24728df94dbdf94115.jpg temp/1743291990_956648_1348696058_1e2533c884c4d1ff31da5e84c41c956a.jpg temp/1743291990_956648_1348695930_0fef44892947e40f2be122d1edbab122.jpg temp/1743291990_956648_1348695831_87177c30fd69e53570f4bf6c24ba86da.jpg temp/1743291990_956648_1348695745_ffd42410a4053010bd85d17218aa1075.jpg temp/1743291990_956648_1348695667_d96e4feca1d4918d5b1aa85f0d9931c1.jpg temp/1743291990_956648_1348694394_9f70bf909abc51f061a6391bff82691c.jpg temp/1743291990_956648_1348694383_a9d59e4b9fccd38c3c32f6e4b0786286.jpg temp/1743291990_956648_1348694380_4137edcd66c6c2f2a64de3b0cacf7cbc.jpg temp/1743291990_956648_1348694376_0cae754ace3cedfb99a922da31d69e7a.jpg temp/1743291990_956648_1348694365_726445ff631f6ad25108734f39c20f57.jpg temp/1743291990_956648_1348694339_7f1cf0345cad512a85e9ece929dcd58c.jpg temp/1743291990_956648_1348694239_a1dff5d1dc9bf601f620434c9f107a2d.jpg temp/1743291990_956648_1348694190_3fe1ab6d140b2d7a7c8edaaf1d2ee595.jpg temp/1743291990_956648_1348694031_a43341b567c432b1889852b28ff51cfc.jpg temp/1743291990_956648_1348693977_0e2248dc6a303dd5ddd795c8a91629b8.jpg temp/1743291990_956648_1348693674_31e5f811bfde2cedc234b4b30375a528.jpg temp/1743291990_956648_1348693640_6a68191c40db7f81e1b97f9d975c653f.jpg temp/1743291990_956648_1348693601_3676a99698a1bac0e8e68d8c661b3bee.jpg temp/1743291990_956648_1348693522_c292424683e553a77a822cda013810fa.jpg temp/1743291990_956648_1348693402_f10460c4238c71a9b8a64194003234d2.jpg temp/1743291990_956648_1348693314_e00f17c9697e976bf12318ec28f24045.jpg temp/1743291990_956648_1348693268_c92ac64417de8b1df8324e8327c15c54.jpg temp/1743291990_956648_1348693235_ce5824845d8969fe0f029302a78ce6d0.jpg temp/1743291990_956648_1348692948_4ba55fa58bcb8eb42bd48e5959503416.jpg temp/1743291990_956648_1348692920_37cf1e5ed4dee995ae4dbfe1aed0f821.jpg temp/1743291990_956648_1348692888_b767c796d12ee952455021dddb7b993d.jpg temp/1743291990_956648_1348692878_3d53d445cfb40afdf59468e1a4588cb7.jpg temp/1743291990_956648_1348692854_d6dbd3a2f388cfbcbc35eabb998b0dd3.jpg temp/1743291990_956648_1348692735_69cd9cf6cfd9a8862089284b22de3414.jpg temp/1743291990_956648_1348692529_44d8040fe5c84d298431e951b02dfc06.jpg temp/1743291990_956648_1348692472_ce22b4af0f744549fde5b9b9f4f26960.jpg temp/1743291990_956648_1348692420_2563908070681a6547a98aec85977c18.jpg temp/1743291990_956648_1348692339_6a9f6a7041245e27ac171c6ecff96170.jpg Found 98 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3146, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1348703354, 1348703328, 1348703311, 1348703309, 1348703306, 1348703304, 1348703301, 1348703299, 1348703295, 1348703267, 1348703045, 1348702708, 1348702152, 1348702127, 1348701908, 1348701906, 1348701905, 1348701896, 1348701801, 1348701763, 1348701718, 1348701669, 1348701616, 1348701574, 1348701505, 1348701497, 1348701414, 1348701386, 1348701346, 1348701286, 1348700948, 1348700871, 1348700838, 1348700811, 1348700784, 1348700742, 1348700492, 1348700460, 1348700384, 1348700314, 1348700189, 1348699349, 1348699241, 1348698850, 1348698749, 1348698642, 1348698576, 1348698362, 1348698289, 1348698196, 1348698159, 1348698101, 1348698097, 1348697851, 1348697790, 1348697746, 1348697718, 1348697679, 1348697558, 1348697463, 1348697462, 1348697460, 1348697449, 1348697422, 1348696956, 1348696058, 1348695930, 1348695831, 1348695745, 1348695667, 1348694394, 1348694383, 1348694380, 1348694376, 1348694365, 1348694339, 1348694239, 1348694190, 1348694031, 1348693977, 1348693674, 1348693640, 1348693601, 1348693522, 1348693402, 1348693314, 1348693268, 1348693235, 1348692948, 1348692920, 1348692888, 1348692878, 1348692854, 1348692735, 1348692529, 1348692472, 1348692420, 1348692339] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 98 time used for this insertion : 0.06133627891540527 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 13.447388648986816 time spend to save output : 0.06558036804199219 total time spend for step 0 : 13.512969017028809 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 12.17user 4.44system 2:34.57elapsed 10%CPU (0avgtext+0avgdata 1998220maxresident)k 676080inputs+83272outputs (2391major+546192minor)pagefaults 0swaps