Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 20.189618349075317 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Mon Apr 7 01:01:49 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 490 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1719 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1719 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1719 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1719 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1500 wait 20 seconds 2025-04-07 01:03:57.269531: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-04-07 01:03:57.297951: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-07 01:03:57.298223: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-07 01:03:57.300246: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-07 01:03:57.301868: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-07 01:03:57.302137: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-07 01:03:57.303950: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-07 01:03:57.304880: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-07 01:03:57.308700: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-07 01:03:57.309777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-07 01:03:57.310124: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-04-07 01:03:57.335107: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-04-07 01:03:57.336899: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fd2a4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-04-07 01:03:57.336941: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-04-07 01:03:57.453771: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xeead610 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-04-07 01:03:57.453821: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-04-07 01:03:57.454827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-04-07 01:03:57.454935: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-07 01:03:57.454973: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-07 01:03:57.455004: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-04-07 01:03:57.455034: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-04-07 01:03:57.455064: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-04-07 01:03:57.455095: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-04-07 01:03:57.455125: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-07 01:03:57.456729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-04-07 01:03:57.456795: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-04-07 01:03:57.457732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-04-07 01:03:57.457756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-04-07 01:03:57.457770: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-04-07 01:03:57.459449: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 1500 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.708879470825195 time used to load_weights : 0.58087158203125 0it [00:00, ?it/s] 100it [00:00, 1155.95it/s]2025-04-07 01:04:07.758485: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-04-07 01:04:07.764815: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-04-07 01:04:07.793605: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-04-07 01:04:07.796923: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-04-07 01:04:07.799139: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1743980495_1907866_1350144312_063f4ee775b2d0834feb1c3631ac30dc.jpg temp/1743980495_1907866_1350144308_a437450a7a85333187bb72248f92f168.jpg temp/1743980495_1907866_1350144300_e07868e3681c81909ec0573a896f876a.jpg temp/1743980495_1907866_1350144299_b5536decf0d5e9fae3ddc743e14bcbdf.jpg temp/1743980495_1907866_1350144298_75fe2336c5cb78558f6b7507f54444f1.jpg temp/1743980495_1907866_1350144296_3289ccff75d4b8025747be5232f40734.jpg temp/1743980495_1907866_1350144293_9575efbafaecc39325b0c29c241bda0b.jpg temp/1743980495_1907866_1350144290_193518556fc8189813f6fc75d4af1f68.jpg temp/1743980495_1907866_1350144277_e3837980c85ecd9dfc8743c5a7c83e86.jpg temp/1743980495_1907866_1350144276_c589e7ad48fd9976695be933f84a6d16.jpg temp/1743980495_1907866_1350144273_e481b1c04d79d547421bc2fe7fa44d4d.jpg temp/1743980495_1907866_1350144270_dd908fa63413e6b52f45eb65a26fe7c9.jpg temp/1743980495_1907866_1350144266_030ce47435a9abfed4d038111823bf39.jpg temp/1743980495_1907866_1350144263_c40b5da26195ce2c71da6445879aeedc.jpg temp/1743980495_1907866_1350144254_3e7d63f594b74a9ee77740648c316064.jpg temp/1743980495_1907866_1350144252_d2ecae03f39afa7a2a46a92227a4eb9c.jpg temp/1743980495_1907866_1350144248_38bf27e16932e6915339a737ce637b93.jpg temp/1743980495_1907866_1350144245_7cc6dea204c71ee87bfd28372cf719b9.jpg temp/1743980495_1907866_1350144243_17d9f8694e7958ac0da8af43b37469bc.jpg temp/1743980495_1907866_1350144238_aa9085466e3365be3de73cb66c899f35.jpg temp/1743980495_1907866_1350144228_52cd888800f2945009ebc0340cb09123.jpg temp/1743980495_1907866_1350144224_a87517afcb3f687e00ac03688dffe86d.jpg temp/1743980495_1907866_1350144223_f784277a44f91a9f7a34bbe0bd1449de.jpg temp/1743980495_1907866_1350144220_356e58f7d9ac761757d4ffd1f2e6069e.jpg temp/1743980495_1907866_1350144217_35bf9cc049fafdbfed51ea2fa50c38ae.jpg temp/1743980495_1907866_1350144213_eac5c5136757da04717fc7994714c390.jpg temp/1743980495_1907866_1350144200_48eb29e2312138e4d326b8fecf67ed2b.jpg temp/1743980495_1907866_1350144197_e239e93819a5046233ec91dedf07c07c.jpg temp/1743980495_1907866_1350144195_6431086474e5e32cc9f4225be4015669.jpg temp/1743980495_1907866_1350144194_49ec06f1191563829ccb3d813357a248.jpg temp/1743980495_1907866_1350144190_554aa33c9db4c8902e52a816f7a61223.jpg temp/1743980495_1907866_1350144188_ebbf6382d6fdd047ceaadbab43593941.jpg temp/1743980495_1907866_1350144175_d2e3f2a29c1753a09624e57536e67a1f.jpg temp/1743980495_1907866_1350144171_0f84b9fd8b6d2f3d79a35ecbb8916bc2.jpg temp/1743980495_1907866_1350144168_eadf25d8bb14ef95ae332ec80dda29e8.jpg temp/1743980495_1907866_1350144164_f28fe1f90083597adb8240be77089066.jpg temp/1743980495_1907866_1350144162_c17e254d433083baf65374c3ba679044.jpg temp/1743980495_1907866_1350144160_d507ec480236073d00d154dbda03da26.jpg temp/1743980495_1907866_1350144154_c5023497dcd14a279d4d9368561ab1a5.jpg temp/1743980495_1907866_1350144153_3c05f047473ad3ccea6389e61ba963b7.jpg temp/1743980495_1907866_1350144151_133d03fa974e10c5ebce222e90737fbb.jpg temp/1743980495_1907866_1350144146_f0164027c9b1296af3259e29031267ea.jpg temp/1743980495_1907866_1350144142_92c6629a58b71ed648f0575155cb54b4.jpg temp/1743980495_1907866_1350144140_e9f70a0bf91163bdbc86c647d2222fcc.jpg temp/1743980495_1907866_1350144128_d048103f49f3fd82cc9d0ca49548c943.jpg temp/1743980495_1907866_1350144125_2ad34fa62ef287d0ae1adada52a341fc.jpg temp/1743980495_1907866_1350144123_04df36b9e28ca2b3905989011f3c4545.jpg temp/1743980495_1907866_1350144119_590d463f14c1245ba3c27b2b6b751d6c.jpg temp/1743980495_1907866_1350144116_8d9e219896780665cee8f2a6ca3f8b1e.jpg temp/1743980495_1907866_1350144115_08a7347c0a585e6429ce0dd37d2094b6.jpg temp/1743980495_1907866_1350144107_2c1c81826652b0375c0d9a7256e69924.jpg temp/1743980495_1907866_1350144103_3932d5dbcf1c1a660d28a3e7a8c71bd2.jpg temp/1743980495_1907866_1350144099_751ece3623a4c43e1f8932f3f8ee577a.jpg temp/1743980495_1907866_1350144094_bedc1e0a5ba23fe04967402f161925c9.jpg temp/1743980495_1907866_1350144090_ef5a37a5c1ea5b627392d35fd0952c6e.jpg temp/1743980495_1907866_1350144087_132660595e210f5ca789bda2bb9e7cb5.jpg temp/1743980495_1907866_1350144075_cb4aee8266b55ab46da65257cbde82aa.jpg temp/1743980495_1907866_1350144074_05c43069c8051b2deb87b303dd3cbb56.jpg temp/1743980495_1907866_1350144073_7db8689c2090afed15963f750e50b9d7.jpg temp/1743980495_1907866_1350144070_dd8583ee4316ec059a1d1c55a6451e32.jpg temp/1743980495_1907866_1350144066_37ffd0f11d50511e67f87c090fbc211d.jpg temp/1743980495_1907866_1350144062_b3b898e68d4fb043ee552686088f8aea.jpg temp/1743980495_1907866_1350144045_ccb6a692efed768b05db8ff4bee58be2.jpg temp/1743980495_1907866_1350144040_423ac3ba145c8b1acb9bf6ae3779f0a2.jpg temp/1743980495_1907866_1350144034_f209d427ca51ed2eaed99a540e6ca747.jpg temp/1743980495_1907866_1350144029_d55257121aaa637459efdfda42bdbeb8.jpg temp/1743980495_1907866_1350144026_0104a314ad2f79df4e711a04f7066f2e.jpg temp/1743980495_1907866_1350144025_e9e1f01f5a2e8d26709eff66a858812f.jpg temp/1743980495_1907866_1350144018_e76ce21c7477fe373f86310b9637a721.jpg temp/1743980495_1907866_1350144014_44ee653ae9fb3a50371866b5b5a4f720.jpg temp/1743980495_1907866_1350144010_24a3d750235f00a7dc2d41386078dece.jpg temp/1743980495_1907866_1350144005_66d500b8b894249829632ef45ec3c502.jpg temp/1743980495_1907866_1350144002_4b4c6cb26a7d19874d4408e6fad7ef94.jpg temp/1743980495_1907866_1350143999_1e9d3f1bcb9f70feb4067bd699ab5ba7.jpg temp/1743980495_1907866_1350143990_02d05a5426909902582380c781fcebb3.jpg temp/1743980495_1907866_1350143989_dbef896be0de373985f2338aca6c5c2f.jpg temp/1743980495_1907866_1350143988_4ebfa9225e98173477aabe71ae48390b.jpg temp/1743980495_1907866_1350143985_b68c43461016388fe16907a4147879c9.jpg temp/1743980495_1907866_1350143982_76bb5be611fd88072d67713242f2c8f3.jpg temp/1743980495_1907866_1350143978_433c83a1f684051d4d2380fa4272f3fa.jpg temp/1743980495_1907866_1350143960_ce129d0a0182424ed260785abf7e047d.jpg temp/1743980495_1907866_1350143956_387e7c97658f43b313da54e3565817e5.jpg temp/1743980495_1907866_1350143952_b1f7f7352c860253d708083bd873f63f.jpg temp/1743980495_1907866_1350143949_fef12ec16bffc984929e3d4bb5358a9c.jpg temp/1743980495_1907866_1350143946_ff37c896b5f46a3552ddf23b2e46e9da.jpg temp/1743980495_1907866_1350143945_dad445afbf2e333fff3e39b45253ef77.jpg temp/1743980495_1907866_1350143935_e354d2abb653f55366511d894e86dd6d.jpg temp/1743980495_1907866_1350143930_f2b5761a79bf70a89c2148f235b32fd6.jpg temp/1743980495_1907866_1350143924_4338cba9e87963f30589151dff593ee1.jpg temp/1743980495_1907866_1350143918_202315f4722951a209b4750f55e11257.jpg temp/1743980495_1907866_1350143912_b6e286e373d89b3743f0b4f94e9d6d3a.jpg temp/1743980495_1907866_1350143906_06a8be88a81f8dec39ec25195ff4656b.jpg temp/1743980495_1907866_1350143892_fe8b5893b1889a42463edac3d5c6092d.jpg temp/1743980495_1907866_1350143889_6e06ca08fef017d657c515d3f0ec55c7.jpg temp/1743980495_1907866_1350143887_ff535d9dd7227d6b6e1653c3d98c2b3d.jpg temp/1743980495_1907866_1350143882_1fd139725ffc63eed0ece3eab523ee38.jpg temp/1743980495_1907866_1350143863_796909c2c9a3b80103ad1f997955a4e3.jpg temp/1743980495_1907866_1350143860_2eaef8461f7c764612d9ef708569e577.jpg temp/1743980495_1907866_1350143852_057affc77255849857b78881306d2973.jpg temp/1743980495_1907866_1350143850_71da7d89d39e7fc60c6b6e684a8fb624.jpg Found 100 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3146, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1350144312, 1350144308, 1350144300, 1350144299, 1350144298, 1350144296, 1350144293, 1350144290, 1350144277, 1350144276, 1350144273, 1350144270, 1350144266, 1350144263, 1350144254, 1350144252, 1350144248, 1350144245, 1350144243, 1350144238, 1350144228, 1350144224, 1350144223, 1350144220, 1350144217, 1350144213, 1350144200, 1350144197, 1350144195, 1350144194, 1350144190, 1350144188, 1350144175, 1350144171, 1350144168, 1350144164, 1350144162, 1350144160, 1350144154, 1350144153, 1350144151, 1350144146, 1350144142, 1350144140, 1350144128, 1350144125, 1350144123, 1350144119, 1350144116, 1350144115, 1350144107, 1350144103, 1350144099, 1350144094, 1350144090, 1350144087, 1350144075, 1350144074, 1350144073, 1350144070, 1350144066, 1350144062, 1350144045, 1350144040, 1350144034, 1350144029, 1350144026, 1350144025, 1350144018, 1350144014, 1350144010, 1350144005, 1350144002, 1350143999, 1350143990, 1350143989, 1350143988, 1350143985, 1350143982, 1350143978, 1350143960, 1350143956, 1350143952, 1350143949, 1350143946, 1350143945, 1350143935, 1350143930, 1350143924, 1350143918, 1350143912, 1350143906, 1350143892, 1350143889, 1350143887, 1350143882, 1350143863, 1350143860, 1350143852, 1350143850] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.22481894493103027 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 138.52627635002136 time spend to save output : 0.2341628074645996 total time spend for step 0 : 138.76043915748596 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 12.92user 3.99system 2:44.52elapsed 10%CPU (0avgtext+0avgdata 1733540maxresident)k 405592inputs+103544outputs (47major+562633minor)pagefaults 0swaps