TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 9022>200 please execute split_portfolio.py -i 23429777 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2980365'] with mtr_portfolio_ids : ['23429777'] and first list_photo_ids : [] new path : /proc/953818/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 33.27465224266052 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Fri May 30 02:41:01 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-05-30 02:41:04.572857: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-30 02:41:04.602419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-30 02:41:04.603010: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-30 02:41:04.607298: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-30 02:41:04.609775: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-30 02:41:04.610869: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-30 02:41:04.614018: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-30 02:41:04.616393: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-30 02:41:04.625078: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-30 02:41:04.626723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-30 02:41:04.627228: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-30 02:41:04.651136: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-30 02:41:04.653025: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ff628000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-30 02:41:04.653074: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-30 02:41:04.762733: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1c136e70 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-30 02:41:04.762790: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-30 02:41:04.763843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-30 02:41:04.763935: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-30 02:41:04.763969: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-30 02:41:04.763999: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-30 02:41:04.764030: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-30 02:41:04.764070: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-30 02:41:04.764104: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-30 02:41:04.764135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-30 02:41:04.765544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-30 02:41:04.765625: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-30 02:41:04.766484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-30 02:41:04.766510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-30 02:41:04.766525: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-30 02:41:04.768126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3439 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-30 02:41:11.815559: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-30 02:41:11.816203: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-30 02:41:11.816813: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2629576704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.241367101669312 time used to load_weights : 0.26794886589050293 0it [00:00, ?it/s] 81it [00:00, 807.36it/s]temp/1748565628_953818_1361637200_e344697629806dd5e42e0d1048db347f.jpg temp/1748565628_953818_1361637195_302463dfee5e7cbd6f17c5ddcc781a2f.jpg temp/1748565628_953818_1361637158_a9b09c68377b2c85c7824127b407eeb9.jpg temp/1748565628_953818_1361637157_d0c416760a294eaa2b8dd79151b27b6c.jpg temp/1748565628_953818_1361637155_24277218c4f1cbbde29921418f754e93.jpg temp/1748565628_953818_1361637154_d55fd381a7fe54a6c1edc7415cb38168.jpg temp/1748565628_953818_1361637150_c4ad541d176bcac4f58c5747bb83034f.jpg temp/1748565628_953818_1361637146_dac37a0f6e8b3cfb6567d29d0eb8c435.jpg temp/1748565628_953818_1361637134_7d2155e90525858cef5a37d7b01431f5.jpg temp/1748565628_953818_1361637132_56f1f15bc4e2e9c1caea09b4bfbfc972.jpg temp/1748565628_953818_1361637130_fc3b85f53450c1018bfe76e447de5044.jpg temp/1748565628_953818_1361637127_0afecafba93ff24e8f9b3aa3c02524ba.jpg temp/1748565628_953818_1361637123_cbd72dd9a96c15be24f1170ae98995f9.jpg temp/1748565628_953818_1361637118_dbadc5b200fa3e771b2bacd4b3f3027f.jpg temp/1748565628_953818_1361637078_ff9c0dbc5101740b6fe3b17a3aba6f73.jpg temp/1748565628_953818_1361637074_22bf205e9d05392ce0846f62b00c8472.jpg temp/1748565628_953818_1361637070_abe85213d085d426b1c757ac06546f68.jpg temp/1748565628_953818_1361637067_d6bca9d51e6c2c8c6c31af6347ae0c51.jpg temp/1748565628_953818_1361637061_1f645274eb613f4ccdf34c2b408d2103.jpg temp/1748565628_953818_1361637058_d092b2e93342420dc176aa34b90a985e.jpg temp/1748565628_953818_1361637042_bcb4b863166d2105388c5993ee2e942d.jpg temp/1748565628_953818_1361637037_ef89b3395b2822e29843a73f912f8a4a.jpg temp/1748565628_953818_1361637032_afd47b01a701644f4039470d512f85f3.jpg temp/1748565628_953818_1361637027_d513e512d8a39d260191ddd5b9992287.jpg temp/1748565628_953818_1361637023_339f2b0099b0af96b14b9a506fdbdf41.jpg temp/1748565628_953818_1361637020_c0c10855d540d96419cf929435cfea70.jpg temp/1748565628_953818_1361636993_ccbda50b126af69cc8098637d6546fda.jpg temp/1748565628_953818_1361636987_cacf66945ec3e7419e3a33e9be818e0d.jpg temp/1748565628_953818_1361636942_7d0d54fdf9cecdf9c8f1d6b79cd55ca4.jpg temp/1748565628_953818_1361636911_d122839a10fd3cdd906d716bb3391698.jpg temp/1748565628_953818_1361636908_fe7f3c2acb50b50e8d6a218dbba7e259.jpg temp/1748565628_953818_1361636906_359628a51fc565dd31ac1d605a838489.jpg temp/1748565628_953818_1361636904_30e79601aa92f85e5a9995049da3cb34.jpg temp/1748565628_953818_1361636901_371ce73b76eefb1ad275c8b60b11dfcb.jpg temp/1748565628_953818_1361636898_19beb7eee7ee48ad7532f508b1cbb594.jpg temp/1748565628_953818_1361636889_1b32740d56060220b2b8379b736d5581.jpg temp/1748565628_953818_1361636886_58a9a1acecbceb469c80456bd9fba1e5.jpg temp/1748565628_953818_1361636884_c1cf7de90772e4fb6b89b8325d566f98.jpg temp/1748565628_953818_1361636877_f8e46a3bc4a4d3febd5933bf704b18de.jpg temp/1748565628_953818_1361636872_76a143be41be7f00e3dde2323444fc69.jpg temp/1748565628_953818_1361636866_667cabbac6fad8b5855f80e50fd3ea3f.jpg temp/1748565628_953818_1361636856_8e844324eac27f81cfda329f7fec810b.jpg temp/1748565628_953818_1361636854_88308ba28f742ba01210d7913df46d1f.jpg temp/1748565628_953818_1361636852_e3e0db982365d3db9142ffb9e45b243b.jpg temp/1748565628_953818_1361636850_340ef94d4ab9cc6ac7cc7231ce2c594b.jpg temp/1748565628_953818_1361636847_b1aed9c0578e3cfd0c3cbe58aef46ec0.jpg temp/1748565628_953818_1361636843_e438269cfa882027a4392125df93d443.jpg temp/1748565628_953818_1361636727_d804dad3d7bd8a1df7ff3717343bdd8b.jpg temp/1748565628_953818_1361636724_10c75f2c4b99a0df3dd262face9323d0.jpg temp/1748565628_953818_1361636721_d8244e42ac01e068edc474d3dc0e8456.jpg temp/1748565628_953818_1361636716_9dbb90f705c40920ac9e85f440c331b1.jpg temp/1748565628_953818_1361636704_a5e373fe107f8a176f120e5223c21bb0.jpg temp/1748565628_953818_1361636693_4b5eba2b88ead6bdd6da889034005a59.jpg temp/1748565628_953818_1361636676_5ed6abd17d5aefded1931b9d7ada1ecb.jpg temp/1748565628_953818_1361636671_398dd25047d302a3ab9aa4a801705770.jpg temp/1748565628_953818_1361636666_52d803d11c88cb171e928576603f7ae8.jpg temp/1748565628_953818_1361636661_ffc8c127d245ade8b33e6453cf109f3f.jpg temp/1748565628_953818_1361636657_4d6f6ef296521e811cab48a029e28b9b.jpg temp/1748565628_953818_1361636654_a118302b6934469e488c4324d10ccf54.jpg temp/1748565628_953818_1361636624_d179ef65df4c49e74b6c7f98f4c9fbcc.jpg temp/1748565628_953818_1361636621_0b2bb3b9382a0ec065b76e057e0461c1.jpg temp/1748565628_953818_1361636619_4b47056b442d734b6898bf167b904a0d.jpg temp/1748565628_953818_1361636617_a43187e37e19d38523faea2b97964ba0.jpg temp/1748565628_953818_1361636615_1581e0b9762a70ac38121b9e12b71d08.jpg temp/1748565628_953818_1361636613_c89f68027d2d828757d8687f338adb3d.jpg temp/1748565628_953818_1361636603_0c2d902bbc68242491e00b33d0efad87.jpg temp/1748565628_953818_1361636598_e530c97bc03ca20ab46f804688d35481.jpg temp/1748565628_953818_1361636594_c790e6c41d03a431ef5990404c03d68f.jpg temp/1748565628_953818_1361636589_195a47d601c317c62c4db3bf31fa124c.jpg temp/1748565628_953818_1361636585_5ca61b10203d7c78394e3b18498e242a.jpg temp/1748565628_953818_1361636581_09fd1b604683ba28b7a1ebfc1d1f54cc.jpg temp/1748565628_953818_1361636549_28c302d9e4ad4ae7fefed1696269ff8b.jpg temp/1748565628_953818_1361636546_74d2c07b7165e43e3c4a97df7c52e9e9.jpg temp/1748565628_953818_1361636544_93b83effefc025e85ecab77e72d90b67.jpg temp/1748565628_953818_1361636541_058415d19005eeb0e35a83d12dfa9efa.jpg temp/1748565628_953818_1361636538_ae29f1f64ede045f80ef7f6f28d408b5.jpg temp/1748565628_953818_1361636535_38820095f2864389554c88e07dd78157.jpg temp/1748565628_953818_1361636522_1df3da2746e75113af12d9887b0e132c.jpg temp/1748565628_953818_1361636518_994d667ba6bbf0dff4320b859ba9163f.jpg temp/1748565628_953818_1361636514_144af6aec8d8d8ea2c2801c6487ce0e3.jpg temp/1748565628_953818_1361636510_a871f9195cfffcadb9c5d2a7685544df.jpg temp/1748565628_953818_1361636505_ec57e3e758913bd0aa72809eadf503b2.jpg temp/1748565628_953818_1361636502_eb045218fc33a28684d726d6fdfbb344.jpg temp/1748565628_953818_1361636496_113cbde6224829b755e47f59fe421f74.jpg temp/1748565628_953818_1361636494_1b19e5ca70cf9c637dcb438136c20375.jpg temp/1748565628_953818_1361636492_78d113275d7061ca22c2d7675ddf3d1b.jpg temp/1748565628_953818_1361636490_84211f46fd6f86ddd10b3d3b459d8bed.jpg temp/1748565628_953818_1361636488_1abe2beed169eaffce7a36523ba57386.jpg temp/1748565628_953818_1361636484_ed5423ccda54e7d43a1216c630a4aa59.jpg temp/1748565628_953818_1361636468_fcc1fbe8de0513dca6ad0cb047204edb.jpg temp/1748565628_953818_1361636463_4e51bca1b1587287d25d5deca7bfd232.jpg temp/1748565628_953818_1361636460_b7830908a0ef61574110d7557ef911d7.jpg temp/1748565628_953818_1361636459_30b06a8248c88d6f6d4b4387ee99888e.jpg temp/1748565628_953818_1361636457_7ca54cddd14c1abec3967a463a0b3f05.jpg temp/1748565628_953818_1361636454_71e3ef5300173a69f8a63b639b268c0c.jpg temp/1748565628_953818_1361636444_89fa836c06a01007c8bda8f7cdf4b4a0.jpg temp/1748565628_953818_1361636442_19ad1f62c9ea4aca4668b1bdd47a8c2f.jpg temp/1748565628_953818_1361636440_6a9521f80cc46566d400b79741e09420.jpg temp/1748565628_953818_1361636437_7d45343fe6db40bdacda07371df876d5.jpg temp/1748565628_953818_1361636433_243eaf8cc5a9b1a57e1a9bb7b7cc2ece.jpg temp/1748565628_953818_1361636429_1237b9cba2a63c17429992e7e8522408.jpg temp/1748565628_953818_1361636420_7eabb42d908200b794b5d02b0533405d.jpg temp/1748565628_953818_1361636417_c360577856d19de69e9d7e8487efa311.jpg temp/1748565628_953818_1361636415_7f5d2416a934947e5f0ebbc7c4f1ee0a.jpg temp/1748565628_953818_1361636412_96f29fde8f171960ffd180a32af816a0.jpg temp/1748565628_953818_1361636408_fcfa6809cb561896cde0006fbb72cc83.jpg temp/1748565628_953818_1361636405_822262ed3a5b98371107631c98540688.jpg temp/1748565628_953818_1361636384_741a9919e526c91bfdaf27f6f1633437.jpg temp/1748565628_953818_1361636382_786995db0f0bf87d2f9877a6f26935bb.jpg temp/1748565628_953818_1361636379_8dc040b32939b0eeeaf50a64c3fb6673.jpg temp/1748565628_953818_1361636376_7e4054090dc9688aa8cd81fd5e272a86.jpg temp/1748565628_953818_1361636375_fb5c5381153cb08287659b5b1d354bb0.jpg temp/1748565628_953818_1361636373_2b1ee605510affeabdf392bae5ae171a.jpg temp/1748565628_953818_1361636365_3db5a251f3300a7e19f1ad4becd353fb.jpg temp/1748565628_953818_1361636360_54fd9fcbd3ebd11b419a7dbcb88013d0.jpg 164it [00:00, 819.61it/s] 200it [00:00, 897.89it/s]2025-05-30 02:41:15.767801: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-30 02:41:15.980698: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-30 02:41:16.082496: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-30 02:41:16.106574: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748565628_953818_1361636356_003708fd12b4fa4b48f474232735d6a8.jpg temp/1748565628_953818_1361636352_0181aa0d55a3b351eccddb0ff133a756.jpg temp/1748565628_953818_1361636349_0c0560a47d75abb077b207b411b3ff3d.jpg temp/1748565628_953818_1361636345_ed2961e14056cdaf0808543f6be4fa54.jpg temp/1748565628_953818_1361636340_60a58bb8f44b8bb659fe4c2c90b27b57.jpg temp/1748565628_953818_1361636337_015af4bfadfa81a6e64c864255ddb0b8.jpg temp/1748565628_953818_1361636334_0121f09b897bac8d4385cb329c92760f.jpg temp/1748565628_953818_1361636330_1c05b5cec684ddedf522ef2b2891bdbb.jpg temp/1748565628_953818_1361636326_e46f0852dd355b03eddb08db1f1bf64e.jpg temp/1748565628_953818_1361636322_7efb1adde79a1434a6b21f153c764dfe.jpg temp/1748565628_953818_1361636281_08c23da444192693e99a398f9f12708c.jpg temp/1748565628_953818_1361636278_84c89d9f7ed34b5040ad5c6c85269d55.jpg temp/1748565628_953818_1361636276_41e5cf88ed79234eb50ac9854e4bfad2.jpg temp/1748565628_953818_1361636274_17c06deb5204780bb058a87179b22400.jpg temp/1748565628_953818_1361636264_62a1e0c154e4818a82a99f95001eb060.jpg temp/1748565628_953818_1361636261_cb5dd8f04ecbdcbb5a494358d18af16d.jpg temp/1748565628_953818_1361636247_38595c2d4bda05fc2d6ce090041a8e46.jpg temp/1748565628_953818_1361636245_27b561a18b6063ab15cfb3a399761bc1.jpg temp/1748565628_953818_1361636244_dadfdaf2bcd8a128ca78a41e6e2147b8.jpg temp/1748565628_953818_1361636241_301be77e4bca0abc724f734718e08b44.jpg temp/1748565628_953818_1361636236_eeb7f6bb95f11fd4af912482654ec3e7.jpg temp/1748565628_953818_1361636231_30fe3457d479df2a481b52d568d6732a.jpg temp/1748565628_953818_1361636218_4ba06064bfe8aa2066f1806ae8f7172d.jpg temp/1748565628_953818_1361636214_9c7eca75ea5a68698afecebc075bf86b.jpg temp/1748565628_953818_1361636210_ddee3acda3502bbd1efd48152b3e94d5.jpg temp/1748565628_953818_1361636202_95e0846558cc7edb52c4107eaa10bd19.jpg temp/1748565628_953818_1361636163_dadbce3335b4e70df3a1a5100b79844f.jpg temp/1748565628_953818_1361636162_0f9899cb78de04533a5ded3c9f32673f.jpg temp/1748565628_953818_1361636161_ecbd91ec38d57a01166fea71d1c902e4.jpg temp/1748565628_953818_1361636160_8c693ff90505d643f8e6ce1cc9183e85.jpg temp/1748565628_953818_1361636126_9cf8a09a6d68cf0ea194f9c380200664.jpg temp/1748565628_953818_1361636123_a3b3a204291ad6fba385666a3a73c2e0.jpg temp/1748565628_953818_1361636121_98bf2c1b2c9917eaa02d9070d970bcfa.jpg temp/1748565628_953818_1361636118_0387907bc09677993af40d45989fa6e0.jpg temp/1748565628_953818_1361636116_4cfbae0b6b69c1fe5de82d16c764ab58.jpg temp/1748565628_953818_1361636114_eaf37686e0d0d76278d7587eb348c260.jpg temp/1748565628_953818_1361636051_dbe4ef04d103343b847c7e35a3c42df3.jpg temp/1748565628_953818_1361636046_d8bd717a22bedabf9db92b000898370c.jpg temp/1748565628_953818_1361636042_ef3e56c9db307a17164dc77bc7f47a89.jpg temp/1748565628_953818_1361636039_b4c5cc54c3d28163da90834ca09ab92b.jpg temp/1748565628_953818_1361636035_7b27c5bff9285ff343846e6cbdf258a1.jpg temp/1748565628_953818_1361636029_1a27fafe651bacbf7b63edf6b1110d47.jpg temp/1748565628_953818_1361636008_0ae364c3bc4fdce66992fb81b1fbcaa6.jpg temp/1748565628_953818_1361636005_71c2f925641a9df68c2992cce8e61bd0.jpg temp/1748565628_953818_1361636003_d0d6ae74c217e2edf4562928b40bd768.jpg temp/1748565628_953818_1361635998_ea3bf6356f6d879ac203b65aa965a748.jpg temp/1748565628_953818_1361635992_18cb72a18f0abe13914424731bbd4e17.jpg temp/1748565628_953818_1361635974_19a10de6aa25a295e1dea50ab34f9de4.jpg temp/1748565628_953818_1361635972_0ef7e6a241ab3ae166d755b86db719dc.jpg temp/1748565628_953818_1361635971_a72dd30d5eafde94f0472379a3f08274.jpg temp/1748565628_953818_1361635970_e705792157cc6e688d3140f93275adf7.jpg temp/1748565628_953818_1361635967_29e7337f9620853e1d718dbf824bf6c9.jpg temp/1748565628_953818_1361635964_08e19f72eae433269ad30bc298e5ac38.jpg temp/1748565628_953818_1361635949_dfc9eea10ccf6c4577db08e4b32c992d.jpg temp/1748565628_953818_1361635945_5df08a61ce19123eaf6b9da06d7b592c.jpg temp/1748565628_953818_1361635942_cf281c00a8c6501e74eb751d46e246bc.jpg temp/1748565628_953818_1361635937_afa8f22437964ac16d06b66be22d99c2.jpg temp/1748565628_953818_1361635933_0dac69c5ad7c05406f980ad69edb9151.jpg temp/1748565628_953818_1361635929_28e637ff9dffa7497814cc2c1ddb9d34.jpg temp/1748565628_953818_1361635924_6a99d1ec82f3a6538f973fe4b20cb498.jpg temp/1748565628_953818_1361635922_69775423a0e5b169bbf36b2a45c5e333.jpg temp/1748565628_953818_1361635921_4d3f279298329c88d74c97736f58020f.jpg temp/1748565628_953818_1361635918_c26d956156a5cb179f21e9730fd1aa0b.jpg temp/1748565628_953818_1361635914_4c76f2a0f18e620ab70578d8837a5ffe.jpg temp/1748565628_953818_1361635910_696886d85cf22eabcd298e29630705d4.jpg temp/1748565628_953818_1361635893_6cd5c5b1715e4e05560757cb611b5ccd.jpg temp/1748565628_953818_1361635891_c73baae73c99fcedba5ec3a7043ed181.jpg temp/1748565628_953818_1361635890_d5637236fafe360b1bcea171c99bd537.jpg temp/1748565628_953818_1361635889_10581f6f02e4548263d6310bc9da31a6.jpg temp/1748565628_953818_1361635888_e3f3d752ba482dffb0a25c2bf28491d7.jpg temp/1748565628_953818_1361635886_a5fcee00bd4c9496e73540b05d744250.jpg temp/1748565628_953818_1361635784_512bf5309bd49d542445173e381d61a3.jpg temp/1748565628_953818_1361635779_50de8b3c8175b7a6473ac76197c4878f.jpg temp/1748565628_953818_1361635771_f70c80fbc55de3588cf9f0c7b24176f7.jpg temp/1748565628_953818_1361635765_8c0e1aa44e447afc0bcd5257031b0dea.jpg temp/1748565628_953818_1361635759_0396a958ad5b9c0669c4ea75891206d9.jpg temp/1748565628_953818_1361635748_e72075b888b3a223307e57eb829cd503.jpg temp/1748565628_953818_1361635647_f2af1a328a49396db15947723a40df88.jpg temp/1748565628_953818_1361635644_0b18e7a92ea4a15689b7b43c23bf631a.jpg temp/1748565628_953818_1361635642_bc9ab3ea6717d8c13534c8064bcecc8a.jpg temp/1748565628_953818_1361635639_0d49fe1139cef6690021c0e909ccaf12.jpg temp/1748565628_953818_1361635636_12f1fb56c6d600895c2bc839b0e0b5e1.jpg temp/1748565628_953818_1361635633_69b170ee0303ff4ade4ef92830b03807.jpg temp/1748565628_953818_1361635628_3a46747b52af926201e601c905383c89.jpg temp/1748565628_953818_1361635625_fa48d2a6f80431af8eb2bb4234b92bc4.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1361637200, 1361637195, 1361637158, 1361637157, 1361637155, 1361637154, 1361637150, 1361637146, 1361637134, 1361637132, 1361637130, 1361637127, 1361637123, 1361637118, 1361637078, 1361637074, 1361637070, 1361637067, 1361637061, 1361637058, 1361637042, 1361637037, 1361637032, 1361637027, 1361637023, 1361637020, 1361636993, 1361636987, 1361636942, 1361636911, 1361636908, 1361636906, 1361636904, 1361636901, 1361636898, 1361636889, 1361636886, 1361636884, 1361636877, 1361636872, 1361636866, 1361636856, 1361636854, 1361636852, 1361636850, 1361636847, 1361636843, 1361636727, 1361636724, 1361636721, 1361636716, 1361636704, 1361636693, 1361636676, 1361636671, 1361636666, 1361636661, 1361636657, 1361636654, 1361636624, 1361636621, 1361636619, 1361636617, 1361636615, 1361636613, 1361636603, 1361636598, 1361636594, 1361636589, 1361636585, 1361636581, 1361636549, 1361636546, 1361636544, 1361636541, 1361636538, 1361636535, 1361636522, 1361636518, 1361636514, 1361636510, 1361636505, 1361636502, 1361636496, 1361636494, 1361636492, 1361636490, 1361636488, 1361636484, 1361636468, 1361636463, 1361636460, 1361636459, 1361636457, 1361636454, 1361636444, 1361636442, 1361636440, 1361636437, 1361636433, 1361636429, 1361636420, 1361636417, 1361636415, 1361636412, 1361636408, 1361636405, 1361636384, 1361636382, 1361636379, 1361636376, 1361636375, 1361636373, 1361636365, 1361636360, 1361636356, 1361636352, 1361636349, 1361636345, 1361636340, 1361636337, 1361636334, 1361636330, 1361636326, 1361636322, 1361636281, 1361636278, 1361636276, 1361636274, 1361636264, 1361636261, 1361636247, 1361636245, 1361636244, 1361636241, 1361636236, 1361636231, 1361636218, 1361636214, 1361636210, 1361636202, 1361636163, 1361636162, 1361636161, 1361636160, 1361636126, 1361636123, 1361636121, 1361636118, 1361636116, 1361636114, 1361636051, 1361636046, 1361636042, 1361636039, 1361636035, 1361636029, 1361636008, 1361636005, 1361636003, 1361635998, 1361635992, 1361635974, 1361635972, 1361635971, 1361635970, 1361635967, 1361635964, 1361635949, 1361635945, 1361635942, 1361635937, 1361635933, 1361635929, 1361635924, 1361635922, 1361635921, 1361635918, 1361635914, 1361635910, 1361635893, 1361635891, 1361635890, 1361635889, 1361635888, 1361635886, 1361635784, 1361635779, 1361635771, 1361635765, 1361635759, 1361635748, 1361635647, 1361635644, 1361635642, 1361635639, 1361635636, 1361635633, 1361635628, 1361635625] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.09229636192321777 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 14.800768375396729 time spend to save output : 0.09847450256347656 total time spend for step 0 : 14.899242877960205 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.44user 4.27system 0:52.91elapsed 35%CPU (0avgtext+0avgdata 1924472maxresident)k 511184inputs+310384outputs (633major+605618minor)pagefaults 0swaps