FBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFB[Errno 104] Connection reset by peer can't download the photo : 1358975672 complete_path : /home/admin/workarea/git/Velours/python/temp/1747402649_382414_1358975672_b4a2842fc514dc24dc23d857493a9fdb.jpg FBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 1 photos in the step downloads : photo missing : [1358975672] try to delete the photos missing in DB Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1358975672 length of list_filenames : 99 ; length of list_pids : 99 ; length of list_args : 99 time to download the photos : 14.627363204956055 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Fri May 16 15:37:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory inside check gpu memory 2025-05-16 15:37:51.703886: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-16 15:37:51.736849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-16 15:37:51.737829: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 15:37:51.762414: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 15:37:51.765588: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-16 15:37:51.767810: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-16 15:37:51.771431: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-16 15:37:51.772650: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-16 15:37:51.777132: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 15:37:51.778169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-16 15:37:51.778656: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-16 15:37:51.803144: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-16 15:37:51.805275: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f6e98000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-16 15:37:51.805336: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-16 15:37:51.919512: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xb4ce800 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-16 15:37:51.919559: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-16 15:37:51.920712: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-16 15:37:51.920812: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 15:37:51.920843: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 15:37:51.920871: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-16 15:37:51.920899: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-16 15:37:51.920926: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-16 15:37:51.920954: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-16 15:37:51.920982: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 15:37:51.922369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-16 15:37:51.922437: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 15:37:51.923321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-16 15:37:51.923348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-16 15:37:51.923362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-16 15:37:51.924769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2944 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-16 15:37:58.909141: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-16 15:37:58.909826: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.678295850753784 time used to load_weights : 0.26552724838256836 0it [00:00, ?it/s] 88it [00:00, 877.12it/s] 99it [00:00, 879.83it/s]2025-05-16 15:38:03.075242: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 15:38:03.341346: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 15:38:03.660924: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-16 15:38:03.683637: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1747402649_382414_1358979961_0e0b2514c8f08ee804f84ca1be2b1f27.jpg temp/1747402649_382414_1358979941_18085720ec54784be605a57a5166043b.jpg temp/1747402649_382414_1358979936_b675f87d067e0a085dc2d90688f5b822.jpg temp/1747402649_382414_1358979897_eaf7c155f87d96f0be1d44e7a8404022.jpg temp/1747402649_382414_1358979589_4e13a482c8b68a1bb6cca4ed723f3f79.jpg temp/1747402649_382414_1358979582_fa6476e42eae635586f42930cb2cbae7.jpg temp/1747402649_382414_1358979360_296f4b6a84f7ed70adf37de2f28ee49b.jpg temp/1747402649_382414_1358979349_3da7a7b4f2e25ce5e5e316f4c701ad12.jpg temp/1747402649_382414_1358978672_27e6c48a8a7021ce769f38d83f0a40c5.jpg temp/1747402649_382414_1358978626_71a0e422b747400183b9f7dca21ae744.jpg temp/1747402649_382414_1358978584_5adc9f863d223962559586ea73d8bf12.jpg temp/1747402649_382414_1358978541_72173f98a0e93c04ea077a1b855aafce.jpg temp/1747402649_382414_1358978452_3dce4cc7287129712ec89a3c9d619b7d.jpg temp/1747402649_382414_1358978447_be6b1e7b64a5a2ab986d37cfbafa6232.jpg temp/1747402649_382414_1358978439_d80c5b3f8d315a849020621050dedeb9.jpg temp/1747402649_382414_1358978434_0f1f4cba3a648e9e00f2e9ce376760c5.jpg temp/1747402649_382414_1358978362_9821bff306ea09349988d083a7990732.jpg temp/1747402649_382414_1358978182_65f7cac3df0a5a28d21a0d16c51b5861.jpg temp/1747402649_382414_1358978173_79f96d3c8aad6c0910369e85d962e8bf.jpg temp/1747402649_382414_1358978077_ad2d6e9a31a1dbf4be10c1383e9f5a08.jpg temp/1747402649_382414_1358977850_60ce54a314a586b01cea62c8de01bb16.jpg temp/1747402649_382414_1358977840_5953b0f2a1dce3324fb7afa5660410c1.jpg temp/1747402649_382414_1358977834_469b1fcbf92b38c2a3ea278fd8143ac4.jpg temp/1747402649_382414_1358977625_bc9a1b3510f9245741b435542717f62d.jpg temp/1747402649_382414_1358977539_e11aef8e8495823037041c3281ac06fb.jpg temp/1747402649_382414_1358977466_191a99c46dea24cb56e4ff83a1a29567.jpg temp/1747402649_382414_1358977181_ee323f95f47bb1825b1a1acf439a30a6.jpg temp/1747402649_382414_1358977170_1c241a143be677cf15405c443c269dc8.jpg temp/1747402649_382414_1358977135_e59aec206a9517a4bc2891fa80bcf974.jpg temp/1747402649_382414_1358976639_737ee231307872385484a780d9651e12.jpg temp/1747402649_382414_1358976635_025b349d58abff8345345959fb319286.jpg temp/1747402649_382414_1358976594_b4fb1918f287e0d7c42cc8c4ba0aa894.jpg temp/1747402649_382414_1358976346_5f42197115c10887a6a30edba236a8cc.jpg temp/1747402649_382414_1358976299_503437f24acddbab371c4e094a315cab.jpg temp/1747402649_382414_1358976223_053cccd63847bc2db7e63b0a389c121c.jpg temp/1747402649_382414_1358976150_438a3d44231796852eab8c45f8c2c9d6.jpg temp/1747402649_382414_1358975969_91dff4445cd7ab4a2dd2b4dc712f2dab.jpg temp/1747402649_382414_1358975952_2d050a7750aa02aef5988cc49e6970ae.jpg temp/1747402649_382414_1358975945_994d661ed87af5f7a41ed2246287f545.jpg temp/1747402649_382414_1358975757_85e93437a8d0a0b36c513cae0c500fc1.jpg temp/1747402649_382414_1358975730_83bf941665c206105b99236f4aa7af18.jpg temp/1747402649_382414_1358975716_399185add216cb486e3e13c4efe9c815.jpg temp/1747402649_382414_1358975595_4821c03857042b8f40c6c74d3956aaec.jpg temp/1747402649_382414_1358975592_501173e0be7d300734f1d16039d90aa4.jpg temp/1747402649_382414_1358975589_661e615f5cbe93585f180f5131b4489a.jpg temp/1747402649_382414_1358975583_f17939c1ef2e2cffca7cbe99ef55d83e.jpg temp/1747402649_382414_1358975578_4adcb6abf6e2bba1e5cdb2bf45ae1b26.jpg temp/1747402649_382414_1358975571_95276e21f93f230bb0bf97d658b495ad.jpg temp/1747402649_382414_1358975253_7d9228d6486b54e4ab808d22df987883.jpg temp/1747402649_382414_1358975247_9d52d6eb8e59ffff553361ccff9a4821.jpg temp/1747402649_382414_1358975243_cd5a787e6fb7e38a9b56c0e46a724a4e.jpg temp/1747402649_382414_1358975170_f219d17d323243079535031f3a67840d.jpg temp/1747402649_382414_1358975166_d65263ecd8919a98b8463ecf2d928962.jpg temp/1747402649_382414_1358975160_b411617a1cc562d2f712d3ee87ae1da4.jpg temp/1747402649_382414_1358974980_6232cddcdd6664495356c9aa60ef43bd.jpg temp/1747402649_382414_1358974970_f7c0b3658ecb8e011d85ec11cda4f1a5.jpg temp/1747402649_382414_1358974920_a0fec6e83a948124c18ec053cbbb3f47.jpg temp/1747402649_382414_1358974892_fa3da49a73f63eba0a7f28eeabda4ddb.jpg temp/1747402649_382414_1358974886_e4984d0b0ca301a7601e8e0bbc6e3f58.jpg temp/1747402649_382414_1358974838_51c1b68df43df64761a0b23cc2e66906.jpg temp/1747402649_382414_1358974540_539b392e723ba8449e6f0a5fbb4cb44b.jpg temp/1747402649_382414_1358974471_a86b53ab7e04fba7af4b935ac56e0107.jpg temp/1747402649_382414_1358974428_ab5fd3eff533ab1232ad1c48c16de2ed.jpg temp/1747402649_382414_1358974217_04109f89ca8b32ea3b7e32f0263d8999.jpg temp/1747402649_382414_1358974050_96880ade8ec960a5e223fef98dfe15b9.jpg temp/1747402649_382414_1358974017_971e2d192ed72be0486fa6bd00961284.jpg temp/1747402649_382414_1358974009_8343708efed8a4a9c036451204c5b8c2.jpg temp/1747402649_382414_1358974004_6f1c399b0260a5cbf2ffc3871c594b5f.jpg temp/1747402649_382414_1358974000_292066e7f04c6b1aabb7d57d1436b385.jpg temp/1747402649_382414_1358973991_fb2966b14c09029b4f39e1b3496efccc.jpg temp/1747402649_382414_1358973488_af97b08d2448cb5cae0c48e38587f423.jpg temp/1747402649_382414_1358973448_8b4291b2feb289a3bb685ea0055297ae.jpg temp/1747402649_382414_1358973435_e57aa673a2c2f00fbd4e3ad311087279.jpg temp/1747402649_382414_1358973432_019a814e27a697f24cbddea8db0c0bb3.jpg temp/1747402649_382414_1358973428_bf949bc766f44979ae07caff21265e66.jpg temp/1747402649_382414_1358973425_8e0dba417ae52f25ea9b99cc15b6450f.jpg temp/1747402649_382414_1358973258_7d13dfa245aef3c8db5913fef6ae76e5.jpg temp/1747402649_382414_1358973217_18adcc7f48384ca2c665a7c68e4d791d.jpg temp/1747402649_382414_1358973177_ea73b98b187c6d2d836034cffa70a9de.jpg temp/1747402649_382414_1358973133_89f0593d84097c7af5476b09caf01254.jpg temp/1747402649_382414_1358973089_a216b95d3226468c4abd4682776f7231.jpg temp/1747402649_382414_1358973049_35a41662da43ceb1f58a01e11ace17e2.jpg temp/1747402649_382414_1358972605_30b71fdce16e98707e7055c2f17f7888.jpg temp/1747402649_382414_1358972597_07155e15c2eb5030d876c5f06078695b.jpg temp/1747402649_382414_1358972588_ac508caa73609ee3f0ff23037fd411b4.jpg temp/1747402649_382414_1358972584_83d263075adedcce9dc8c27b7ef1bc6a.jpg temp/1747402649_382414_1358972560_6222e03abcb8ec4227b418c11ec2a5e2.jpg temp/1747402649_382414_1358972539_3c2668e5c3c16f193bc867f80ed504f1.jpg temp/1747402649_382414_1358972233_ce368011f26788e27210ff4aabbe9fb6.jpg temp/1747402649_382414_1358971799_c1fdf3190514ddce337726e71a8e4e0c.jpg temp/1747402649_382414_1358971376_c331af9e4948ca73d18b96878189ba45.jpg temp/1747402649_382414_1358971296_9c0918729115b909f4ea7f4e65fd12fb.jpg temp/1747402649_382414_1358971244_cce69b187b5e9d059fe27ce903307473.jpg temp/1747402649_382414_1358971201_01b11246113a51bb191845e63d0bc83f.jpg temp/1747402649_382414_1358971157_ca1a568bf1e01c70b41e709e0f9ab5c1.jpg temp/1747402649_382414_1358971109_dfd5ee028add10f5c5f9a72d17f6b10a.jpg temp/1747402649_382414_1358970436_7dcbb262e5d91b5b649b1fa81f5fea83.jpg temp/1747402649_382414_1358970376_6d8fc0f2345c0588cbf3a2fb50d92d89.jpg temp/1747402649_382414_1358970278_30ce9e1668ab5d455f0692d08202f9ab.jpg Found 99 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1358979961, 1358979941, 1358979936, 1358979897, 1358979589, 1358979582, 1358979360, 1358979349, 1358978672, 1358978626, 1358978584, 1358978541, 1358978452, 1358978447, 1358978439, 1358978434, 1358978362, 1358978182, 1358978173, 1358978077, 1358977850, 1358977840, 1358977834, 1358977625, 1358977539, 1358977466, 1358977181, 1358977170, 1358977135, 1358976639, 1358976635, 1358976594, 1358976346, 1358976299, 1358976223, 1358976150, 1358975969, 1358975952, 1358975945, 1358975757, 1358975730, 1358975716, 1358975595, 1358975592, 1358975589, 1358975583, 1358975578, 1358975571, 1358975253, 1358975247, 1358975243, 1358975170, 1358975166, 1358975160, 1358974980, 1358974970, 1358974920, 1358974892, 1358974886, 1358974838, 1358974540, 1358974471, 1358974428, 1358974217, 1358974050, 1358974017, 1358974009, 1358974004, 1358974000, 1358973991, 1358973488, 1358973448, 1358973435, 1358973432, 1358973428, 1358973425, 1358973258, 1358973217, 1358973177, 1358973133, 1358973089, 1358973049, 1358972605, 1358972597, 1358972588, 1358972584, 1358972560, 1358972539, 1358972233, 1358971799, 1358971376, 1358971296, 1358971244, 1358971201, 1358971157, 1358971109, 1358970436, 1358970376, 1358970278] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 99 time used for this insertion : 0.09787273406982422 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 20.032687187194824 time spend to save output : 0.10467267036437988 total time spend for step 0 : 20.137359857559204 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 13.97user 4.70system 0:40.38elapsed 46%CPU (0avgtext+0avgdata 2143064maxresident)k 421584inputs+332712outputs (291major+692095minor)pagefaults 0swaps