FBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 35.20419120788574 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu May 29 08:51:03 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-05-29 08:51:06.579159: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-29 08:51:06.609964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-29 08:51:06.610227: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 08:51:06.612384: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 08:51:06.614011: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-29 08:51:06.614291: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-29 08:51:06.616216: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-29 08:51:06.617130: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-29 08:51:06.621150: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 08:51:06.622155: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-29 08:51:06.622495: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-29 08:51:06.651096: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-29 08:51:06.652862: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f2d78000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-29 08:51:06.652905: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-29 08:51:06.750725: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3d396120 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-29 08:51:06.750779: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-29 08:51:06.751870: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-29 08:51:06.751953: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 08:51:06.751983: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 08:51:06.752011: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-29 08:51:06.752038: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-29 08:51:06.752065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-29 08:51:06.752091: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-29 08:51:06.752118: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 08:51:06.753536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-29 08:51:06.753607: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 08:51:06.754466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-29 08:51:06.754491: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-29 08:51:06.754505: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-29 08:51:06.755987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3218 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-29 08:51:13.468986: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.242488622665405 time used to load_weights : 0.38084936141967773 0it [00:00, ?it/s] 79it [00:00, 785.82it/s]temp/1748501428_1370690_1361458439_e593126dfbb823519ae00380e107a4a7_6538465.jpg temp/1748501428_1370690_1361458437_3ace93c1c03d20e5b12a3311c0d66589_78191465.jpg temp/1748501428_1370690_1361458435_aa23909ad099be45d092c83d313cda54_61372249.jpg temp/1748501428_1370690_1361458434_ef3a2c0b30f34f48993f0f9b645ec67d_56798125.jpg temp/1748501428_1370690_1361458424_b18c554816c1cc5197d2da42e34f88ab_66650192.jpg temp/1748501428_1370690_1361458423_11a951002314b6aea2c4cb4c609f19c8_57949936.jpg temp/1748501428_1370690_1361458422_cab9639d01ddc1ab7e056eaa1f684441_44828300.jpg temp/1748501428_1370690_1361458421_7b6fa6f6d266eec15a8ad507d59b0741_82564728.jpg temp/1748501428_1370690_1361458420_38243bdc4bf1d51f484bdd9b8217918b_5441964.jpg temp/1748501428_1370690_1361458419_e13f9b3e8efd7932ea4c35d60104c54d_37987715.jpg temp/1748501428_1370690_1361458412_02817898e29c8a6ede536061c5954435_47115283.jpg temp/1748501428_1370690_1361458411_483aee890c0c13fd3fbc536190589306_88438382.jpg temp/1748501428_1370690_1361458410_0eb64b308c3b4c9188a0f0f758d6d8c4_97634770.jpg temp/1748501428_1370690_1361458409_a924ab177a4500520227f2e9336be1bc_20554158.jpg temp/1748501428_1370690_1361458408_0eaea48b685e69bb4446ad6e2c0ae895_36832512.jpg temp/1748501428_1370690_1361458407_b30db0dbc5997bcb57fa35e0f7043940_87673447.jpg temp/1748501428_1370690_1361458391_bb5e853fd8fd555a6da644a9e7fbbfbd_70068482.jpg temp/1748501428_1370690_1361458390_8621077bfbc74891836ce93213072155_71335301.jpg temp/1748501428_1370690_1361458389_5781fffdceec43da69a9e6a54c2d0fc0_54472113.jpg temp/1748501428_1370690_1361458388_68ca9192c16755e3f85ef57e4414563a_48992350.jpg temp/1748501428_1370690_1361458387_050580255d45b21e8f67fdf60e17fe89_1109998.jpg temp/1748501428_1370690_1361458386_2e5fb18f60b17843bfdc0a464a5938ad_12114632.jpg temp/1748501428_1370690_1361458372_80b4851c46e5e2fdb05b1d250a5e8f35_62877595.jpg temp/1748501428_1370690_1361458371_b9eca541f270018b6501bc138866139e_10780613.jpg temp/1748501428_1370690_1361458370_7a216447692d1ab8439254d5711ea190_6119771.jpg temp/1748501428_1370690_1361458369_b30214e2faef51a413a6692b32d574d1_66244062.jpg temp/1748501428_1370690_1361458368_60d82fad7344f7910b15f66a1a33e068_43571541.jpg temp/1748501428_1370690_1361458367_948dea3c21d9709092c6d4b085880cee_97830884.jpg temp/1748501428_1370690_1361458366_ea2296167dcd592de6d319e76e05bc70_96640620.jpg temp/1748501428_1370690_1361458365_f1f52d5fa17e94e3e51fc7bce8a6e593_63852204.jpg temp/1748501428_1370690_1361458364_e2a5a14bb50ee4d2c23a39b51aa4c3f3_52114140.jpg temp/1748501428_1370690_1361458363_66fb65995b82ccf6573ff3c5d22a4cc8_96931074.jpg temp/1748501428_1370690_1361458362_13bce36ddea042f45eef16745cafc5f5_70711080.jpg temp/1748501428_1370690_1361458361_3f7889c3445ec047f22948e70639f84d_34917064.jpg temp/1748501428_1370690_1361458348_20a68c68eb27b2ddb52db0c80daab52f_77714026.jpg temp/1748501428_1370690_1361458347_557e962927fbd7927ff3abaf17a41d75_92853940.jpg temp/1748501428_1370690_1361458346_47f754cf2650edb8ae4efa9cbb9c029c_84326873.jpg temp/1748501428_1370690_1361458345_b2bfc4c8e88f1836a69f97bf3daa6a85_72669551.jpg temp/1748501428_1370690_1361458344_2059e1bf71417409c765b4d41ff94636_48289307.jpg temp/1748501428_1370690_1361458343_16c5eaedbb4856fb2f5b697a6c1ff747_7035895.jpg temp/1748501428_1370690_1361458341_4896322bcbccad0545f37f0ff52ee0a0_73798497.jpg temp/1748501428_1370690_1361458340_ceded1a3269823cff05e32d58f14e9c7_24485486.jpg temp/1748501428_1370690_1361458339_83f3ad17c6356683276f6c7a4f2a2876_2074834.jpg temp/1748501428_1370690_1361458338_e7ca35f597c7470970eb3344c5ade741_98491132.jpg temp/1748501428_1370690_1361458337_2d2097bcb37f8e03f7e584ba7821da95_6520821.jpg temp/1748501428_1370690_1361458336_fb1f2e7c16506ddbbf2d39b3c8405071_77706624.jpg temp/1748501428_1370690_1361458329_c80795be53a5da561988753908507155_46793602.jpg temp/1748501428_1370690_1361458328_22a6195b14dc6ffcbc4e3a45b79af029_51532488.jpg temp/1748501428_1370690_1361458327_6572aa62348c6927cd45ef4ee93bf680_57982987.jpg temp/1748501428_1370690_1361458326_325ef74c2a83fad44d8affdd984dd27b_25688402.jpg temp/1748501428_1370690_1361458325_748b930047d9c56b571638511add02c6_64617184.jpg temp/1748501428_1370690_1361458324_ce7e6241e72f5af8a088e2caf8a4fcb3_72749847.jpg temp/1748501428_1370690_1361458323_bfc90162702c5a70f28c6b2590df1cb9_82776551.jpg temp/1748501428_1370690_1361458322_cde354cba6b70e3efdb8ea9aaeb36e5e_70200192.jpg temp/1748501428_1370690_1361458321_98710d5982f241aadce3bd7069fc1869_80493933.jpg temp/1748501428_1370690_1361458320_bb1ef3b113113d08d44739356925cb5e_90296217.jpg temp/1748501428_1370690_1361458319_aaea13d1785276140a1af17c43d940f5_63762094.jpg temp/1748501428_1370690_1361458318_0fa864f43fddd2d851b5f75f9d09970e_69587859.jpg temp/1748501428_1370690_1361458303_a9e13ccc3a40f68b7818585c5379536d_84792312.jpg temp/1748501428_1370690_1361458302_51c1735c13ace02433f77aeac49f8921_58307861.jpg temp/1748501428_1370690_1361458301_227a885bf9a755b7d4962966a7633aa9_80099949.jpg temp/1748501428_1370690_1361458300_f95901c8a06979e186e7c8901824a007_67227764.jpg temp/1748501428_1370690_1361458299_fdf80855f8e3294d16c67356c2539e99_68204672.jpg temp/1748501428_1370690_1361458298_3cd50215c4d6d141946fb2c5dbcb8f75_14883928.jpg temp/1748501428_1370690_1361458297_3eeb09420442d8363bf28ae20b8d6db9_21691953.jpg temp/1748501428_1370690_1361458296_35a24803385adb4e8f4a4788d4e64374_47192862.jpg temp/1748501428_1370690_1361458295_6f0ec75fa87b8677e082bc59877a5172_92510035.jpg temp/1748501428_1370690_1361458294_e3036826e67efe38af4b797095212d89_69981161.jpg temp/1748501428_1370690_1361458293_d99add95a562c3b72a21b308f70dbb4b_36845670.jpg temp/1748501428_1370690_1361458292_7115de8005bd28920170fdf8e9ceda1c_44785074.jpg temp/1748501428_1370690_1361458278_4e9a16b22c98d62d535aea7dd3f9cc2f_29399723.jpg temp/1748501428_1370690_1361458277_9bcd36d69c84b9fc9c6bfeb813094000_51987576.jpg temp/1748501428_1370690_1361458276_f010e19d8e49b314be7bc647ab32b596_17252317.jpg temp/1748501428_1370690_1361458275_400971f9601c02849bfd9ea19b33a472_5864438.jpg temp/1748501428_1370690_1361458274_b2552bf3ebf34760e36a86568ddd102d_25558006.jpg temp/1748501428_1370690_1361458272_d1b9b1f34f6710caf51f5d0b9a26fca0_10108497.jpg temp/1748501428_1370690_1361458270_69d54604395da0795fab21e64429a064_90741009.jpg temp/1748501428_1370690_1361458269_73356dd8cc0b41bee8e358d0be6d37f9_57337202.jpg temp/1748501428_1370690_1361458267_b48801ca4a691e3cd9762fa13a0bc605_73902835.jpg temp/1748501428_1370690_1361458265_05f4708214b6917322119a6e03bdc756_42272875.jpg temp/1748501428_1370690_1361458263_cc374e81481282e310d17270373c5db9_27972573.jpg temp/1748501428_1370690_1361458261_5c85576dad18612f3b0f60c3146005ae_64698702.jpg temp/1748501428_1370690_1361458238_cf89aa67f08e11d293c88e8c1141c9e7_95417711.jpg temp/1748501428_1370690_1361458236_9ca3ae3d4ed777f743ec4c7c9601e3c8_19791909.jpg temp/1748501428_1370690_1361458234_1c679fcd8365ad0a2ad0ff83c90b43be_35277689.jpg temp/1748501428_1370690_1361458233_2ca38f19c2bcc16affb070d6243934a2_47619764.jpg temp/1748501428_1370690_1361458232_0a654bb18399bb2041ba199669e12af5_78621861.jpg temp/1748501428_1370690_1361458231_a59af85d931afbc37ccafd35f139056f_34198969.jpg temp/1748501428_1370690_1361458230_c6282234f7ae1051dcca5db06f481057_72392337.jpg temp/1748501428_1370690_1361458229_7d13ed35040186fea6d8de77d571fb92_4876007.jpg temp/1748501428_1370690_1361458228_40fb416c84a14162a1f5f888c8437f54_48452915.jpg temp/1748501428_1370690_1361458227_0cb37798430f771ea5338a2ec2be8f00_75411215.jpg temp/1748501428_1370690_1361458226_4e37415237b8a3f53bfdc875e9fb71fb_37776410.jpg temp/1748501428_1370690_1361458225_23f3ea4c39e2cf9f99a768367a0465ce_52777533.jpg temp/1748501428_1370690_1361458214_3a31adb27e2e0776342930da25ae9099_64086280.jpg temp/1748501428_1370690_1361458213_72664a4ec996723c845519453aab1816_67651096.jpg temp/1748501428_1370690_1361458211_dfae9405bdb92601bf938fac771289f0_91971058.jpg temp/1748501428_1370690_1361458210_b3f8aa922c288d781d65483e1cbfd33a_59923122.jpg temp/1748501428_1370690_1361458209_a8373c78f267dba18249097a65cc04d8_42097610.jpg temp/1748501428_1370690_1361458208_1ebc4e5f225b1178abd0cc9ba7aa2c2a_48702976.jpg temp/1748501428_1370690_1361458206_61517237aced3d16a210626da95e62bd_75316209.jpg 158it [00:00, 736.60it/s] 200it [00:00, 727.42it/s]2025-05-29 08:51:17.694663: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 08:51:17.930054: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 08:51:18.149081: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-29 08:51:18.152083: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748501428_1370690_1361458205_74bcada9438bfe5a295b718f9143042b_98189166.jpg temp/1748501428_1370690_1361458203_a208b26572f7f40182fcc116f07f9a27_18456693.jpg temp/1748501428_1370690_1361458201_250a44d8f6461bd0351e611ddffc8757_54707682.jpg temp/1748501428_1370690_1361458199_0e2dc7551033f62b4f1ec425049ec336_17711458.jpg temp/1748501428_1370690_1361458197_f50b2b8ebbebf7fab8280f1dd351ae15_68938127.jpg temp/1748501428_1370690_1361458183_93d218f9b304aa5e8d7748cf2da1fbb1_43371274.jpg temp/1748501428_1370690_1361458182_f0e566193376071b45be9e69f31a088c_43081341.jpg temp/1748501428_1370690_1361458181_3ba72a0bc2a2208bcf01f68b6f24057c_70414175.jpg temp/1748501428_1370690_1361458180_09dacddd010805043d2c99a916ecd43e_89045136.jpg temp/1748501428_1370690_1361458179_5e83daa88817fca5e5ec143523b06ed3_390847.jpg temp/1748501428_1370690_1361458178_dc4f9d9bb96ee3b5020d5ecc04d0a447_2573749.jpg temp/1748501428_1370690_1361458165_df5f069e53742d59580fa17d078fce08_23069251.jpg temp/1748501428_1370690_1361458163_3bdebb1c7146faea816822e3c7775576_41569334.jpg temp/1748501428_1370690_1361458162_eeec056a3f23bdf3d77d084947801eb3_84177625.jpg temp/1748501428_1370690_1361458161_a56c55e5a4c97a3a16271d26fedf6d0d_88419238.jpg temp/1748501428_1370690_1361458160_5cdc1f933ef480d65e37da864721859b_5890852.jpg temp/1748501428_1370690_1361458159_d73072c56c7d1338f4a7d9ec0aab7cc6_67932232.jpg temp/1748501428_1370690_1361458155_3d7cd7d7253d84888073fdd3d5a6a846_34625093.jpg temp/1748501428_1370690_1361458154_67ff624219eac33c1b2070baccda89ac_94842520.jpg temp/1748501428_1370690_1361458153_66a18d9cd0f8b8d753661137704f697b_61482241.jpg temp/1748501428_1370690_1361458152_7ff5fcba54cd9917abff86d376f1a25d_83875920.jpg temp/1748501428_1370690_1361458151_eb742be439be6b0efdbe32b688ccd981_1012230.jpg temp/1748501428_1370690_1361458150_b142e34dd999c50e610902e2c323ca9c_6639449.jpg temp/1748501428_1370690_1361458137_d34ef302fe91b9601a566a30697adfe5_32378781.jpg temp/1748501428_1370690_1361458136_17dc1087eb9dcbb761f040c367731bc4_52118541.jpg temp/1748501428_1370690_1361458135_ce68bf177fa00f606fe1056688e89246_83765220.jpg temp/1748501428_1370690_1361458134_ad88d683541a5cc0d0df843079287d0f_88763772.jpg temp/1748501428_1370690_1361458133_c909ea5c0b6294160d585117a47a6473_43050135.jpg temp/1748501428_1370690_1361458132_2cbe9c2a06f8b7820a5f46985d4b2744_32927640.jpg temp/1748501428_1370690_1361458129_9e83b932254f07969404aae1106fc863_98769052.jpg temp/1748501428_1370690_1361458127_31151907ff923b80d86bfd618b0fe6fc_54709221.jpg temp/1748501428_1370690_1361458126_9202b3408bfa76d983dbb8c5136afb46_74974881.jpg temp/1748501428_1370690_1361458125_cd94eabf8ea2013f97e39be23561609f_99854032.jpg temp/1748501428_1370690_1361458124_890168d53c397c9f31a66c9bb0d1c670_93010757.jpg temp/1748501428_1370690_1361458123_f13302a392166bc49905637ced2982e9_37529311.jpg temp/1748501428_1370690_1361458119_4b1145b2fb38ec885225110c7856bc6c_30915728.jpg temp/1748501428_1370690_1361458118_766562a72f23a170be882f5b9b1473b2_15419218.jpg temp/1748501428_1370690_1361458117_48683755b6f7a67324318a32da7114cf_57672694.jpg temp/1748501428_1370690_1361458116_cb11b3f78dfec86c8014c9222722ce02_71703481.jpg temp/1748501428_1370690_1361458115_d5b75d8af0b05aa0c4e2454d852d4fff_90936937.jpg temp/1748501428_1370690_1361458114_19c7bafafaeb40cce700e17773a5f4da_85101890.jpg temp/1748501428_1370690_1361458092_522666c6f8f3c139b01f3f3b3b657dee_47861233.jpg temp/1748501428_1370690_1361458091_96d62270fb507516724c54e912cf77ee_77268714.jpg temp/1748501428_1370690_1361458090_06a5a9d03e6bfe6aaafb90e3bddf6e88_37643052.jpg temp/1748501428_1370690_1361458089_6fbcead7646b4059872a9a5372fdc14b_45521974.jpg temp/1748501428_1370690_1361458088_785465da3a09b92fb03e406911d895c3_75169287.jpg temp/1748501428_1370690_1361458087_e2c25fec4542bb2a0377a243817d0966_52379344.jpg temp/1748501428_1370690_1361458086_7039da5040cfbce38272f6bab6de664b_61782450.jpg temp/1748501428_1370690_1361458085_6246e615cee41e931146b74412de5b31_31180154.jpg temp/1748501428_1370690_1361458084_d96c21f3ea9ca6fd51422cf7a3e4108c_49202238.jpg temp/1748501428_1370690_1361458083_0a619317c73733ef67926658a009d7f1_48802711.jpg temp/1748501428_1370690_1361458082_f1fb060743f79818fed27a9ad7739677_67484676.jpg temp/1748501428_1370690_1361458081_6d55e1b8f78766e61818500dd363c6f5_77504745.jpg temp/1748501428_1370690_1361458064_e26836f6f186687297aba646e117a1f4_67022412.jpg temp/1748501428_1370690_1361458063_837ffffb638f7acb936ea708efef0ab6_62866929.jpg temp/1748501428_1370690_1361458062_ea04a6addb89e43ee36f3f0b13d6c3f8_71067749.jpg temp/1748501428_1370690_1361458061_4f54f32d4aa01e97260db0fff8a1d095_96617552.jpg temp/1748501428_1370690_1361458060_e6474329cc2918abe153fd2a3edf4113_11839732.jpg temp/1748501428_1370690_1361458059_f522e4d5a32cbcc2d039d414f2b148aa_13065730.jpg temp/1748501428_1370690_1361458058_bf2abde80a6f8682624279ec04c1c6bd_75557252.jpg temp/1748501428_1370690_1361458057_51e4522701e9cda1913299e8264baf9e_13133445.jpg temp/1748501428_1370690_1361458056_28efbd27ba0cd7e380e35dd8f4e26723_37614260.jpg temp/1748501428_1370690_1361458055_4677c8868928425e9533f48685ad5862_45156727.jpg temp/1748501428_1370690_1361458054_36f66442530f30ae19bee017796355d5_7345274.jpg temp/1748501428_1370690_1361458052_7484e4614968eafa6e43c2eca3569ba3_65903737.jpg temp/1748501428_1370690_1361457966_90e505485629bd719c20a2f204577aab_58357272.jpg temp/1748501428_1370690_1361457965_b8430421edc6385d5ffbcdc9c5cb5bb1_64941091.jpg temp/1748501428_1370690_1361457964_c705dd0988697a52ffa96de25a8981e6_99706950.jpg temp/1748501428_1370690_1361457963_b37543e4588ae896216e7de68c31f8f5_20100126.jpg temp/1748501428_1370690_1361457962_98b2fa21f5be0c382c40d4ab257c8e4e_457831.jpg temp/1748501428_1370690_1361457961_dc3c52ffde7f7a79379627f5c37743a0_47116546.jpg temp/1748501428_1370690_1361457960_c6b890f62de9cb001da8405afc9749ac_74076996.jpg temp/1748501428_1370690_1361457959_fe907f3827ea8747d40760fcf8555488_45807302.jpg temp/1748501428_1370690_1361457958_9bcc6ecf68e0635703652d0b2b4e81ae_69007504.jpg temp/1748501428_1370690_1361457956_cd55da90b6c17a3489d445ca780da58f_64546351.jpg temp/1748501428_1370690_1361457955_21fde9e61261b169f4640905b91d98eb_86216074.jpg temp/1748501428_1370690_1361457954_6c9e810a90cd8b453bb09685f802a49f_6335939.jpg temp/1748501428_1370690_1361457939_01044312b226b70fe4f44d0ef5c11d34_65806060.jpg temp/1748501428_1370690_1361457938_ca889d119e05469bef56871b30ced341_41984505.jpg temp/1748501428_1370690_1361457936_cd36d04ce243b7b60a60ef252d5a31ee_86651079.jpg temp/1748501428_1370690_1361457934_22ba81a5baac3bdae66cb29dd48c5416_86157413.jpg temp/1748501428_1370690_1361457932_b35db7e1cb0fed1ef3455902ba144037_24911338.jpg temp/1748501428_1370690_1361457930_b7485a275748f977268c2981a08b1994_67408482.jpg temp/1748501428_1370690_1361457927_3c174c9d72031a0f5283711156c827fd_57154183.jpg temp/1748501428_1370690_1361457926_eda7932691c48f55b7149f513aa95630_5538733.jpg temp/1748501428_1370690_1361457925_abf6078b91fef2a94bba66089d9fd341_22812749.jpg temp/1748501428_1370690_1361457924_acf32a07283f324b85e318505906c86a_32778937.jpg temp/1748501428_1370690_1361457923_0d864b9fca8c9ff56ab83e5c8e8fc83a_86765580.jpg temp/1748501428_1370690_1361457922_5d8ec619a2314993781e6a2574ca722e_49690972.jpg temp/1748501428_1370690_1361457914_fb601466d75f47666271e950fce1a9a0_73388575.jpg temp/1748501428_1370690_1361457912_9e325380d1bc452c1e28f8bafb977f95_95666597.jpg temp/1748501428_1370690_1361457910_8409a0cfa863476a01900ab8972a5cf8_7606878.jpg temp/1748501428_1370690_1361457908_b598db7b0e6b7415c4c42329a8732b47_15918821.jpg temp/1748501428_1370690_1361457906_400fc16bb01ee6cb0316b66240f5212c_12310492.jpg temp/1748501428_1370690_1361457904_3b73fdf84cdda5693703c825f201511a_7677538.jpg temp/1748501428_1370690_1361457900_7ef48e8d635b9e58685e063016f1b4ee_16003448.jpg temp/1748501428_1370690_1361457899_be0414969cbc178e9a43e1e4e4001705_30831814.jpg temp/1748501428_1370690_1361457898_8eaf9bbac1d84c5292557eb584bb371e_83486865.jpg temp/1748501428_1370690_1361457897_9e9b2cc6875320218bf7ac285508668b_54816750.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1361458439, 1361458437, 1361458435, 1361458434, 1361458424, 1361458423, 1361458422, 1361458421, 1361458420, 1361458419, 1361458412, 1361458411, 1361458410, 1361458409, 1361458408, 1361458407, 1361458391, 1361458390, 1361458389, 1361458388, 1361458387, 1361458386, 1361458372, 1361458371, 1361458370, 1361458369, 1361458368, 1361458367, 1361458366, 1361458365, 1361458364, 1361458363, 1361458362, 1361458361, 1361458348, 1361458347, 1361458346, 1361458345, 1361458344, 1361458343, 1361458341, 1361458340, 1361458339, 1361458338, 1361458337, 1361458336, 1361458329, 1361458328, 1361458327, 1361458326, 1361458325, 1361458324, 1361458323, 1361458322, 1361458321, 1361458320, 1361458319, 1361458318, 1361458303, 1361458302, 1361458301, 1361458300, 1361458299, 1361458298, 1361458297, 1361458296, 1361458295, 1361458294, 1361458293, 1361458292, 1361458278, 1361458277, 1361458276, 1361458275, 1361458274, 1361458272, 1361458270, 1361458269, 1361458267, 1361458265, 1361458263, 1361458261, 1361458238, 1361458236, 1361458234, 1361458233, 1361458232, 1361458231, 1361458230, 1361458229, 1361458228, 1361458227, 1361458226, 1361458225, 1361458214, 1361458213, 1361458211, 1361458210, 1361458209, 1361458208, 1361458206, 1361458205, 1361458203, 1361458201, 1361458199, 1361458197, 1361458183, 1361458182, 1361458181, 1361458180, 1361458179, 1361458178, 1361458165, 1361458163, 1361458162, 1361458161, 1361458160, 1361458159, 1361458155, 1361458154, 1361458153, 1361458152, 1361458151, 1361458150, 1361458137, 1361458136, 1361458135, 1361458134, 1361458133, 1361458132, 1361458129, 1361458127, 1361458126, 1361458125, 1361458124, 1361458123, 1361458119, 1361458118, 1361458117, 1361458116, 1361458115, 1361458114, 1361458092, 1361458091, 1361458090, 1361458089, 1361458088, 1361458087, 1361458086, 1361458085, 1361458084, 1361458083, 1361458082, 1361458081, 1361458064, 1361458063, 1361458062, 1361458061, 1361458060, 1361458059, 1361458058, 1361458057, 1361458056, 1361458055, 1361458054, 1361458052, 1361457966, 1361457965, 1361457964, 1361457963, 1361457962, 1361457961, 1361457960, 1361457959, 1361457958, 1361457956, 1361457955, 1361457954, 1361457939, 1361457938, 1361457936, 1361457934, 1361457932, 1361457930, 1361457927, 1361457926, 1361457925, 1361457924, 1361457923, 1361457922, 1361457914, 1361457912, 1361457910, 1361457908, 1361457906, 1361457904, 1361457900, 1361457899, 1361457898, 1361457897] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 1.0632655620574951 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 14.453384399414062 time spend to save output : 1.0732495784759521 total time spend for step 0 : 15.526633977890015 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.36user 4.67system 0:55.66elapsed 34%CPU (0avgtext+0avgdata 2012080maxresident)k 195344inputs+310592outputs (120major+653706minor)pagefaults 0swaps