python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 129914 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 10161>200 please execute split_portfolio.py -i 24893090 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['3277454'] with mtr_portfolio_ids : ['24893090'] and first list_photo_ids : [] new path : /proc/129914/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBHTTP Error 503: Service Unavailable can't download the photo : 1376730414 complete_path : /home/admin/workarea/git/Velours/python/temp/1755094827_129914_1376730414_a912a6d6e16d6f6912679e18669bdfcd.jpg FBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 1 photos in the step downloads : photo missing : [1376730414] try to delete the photos missing in DB Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1376730414 length of list_filenames : 199 ; length of list_pids : 199 ; length of list_args : 199 time to download the photos : 50.12303185462952 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Wed Aug 13 16:21:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-08-13 16:21:20.228750: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-08-13 16:21:20.261769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-08-13 16:21:20.262083: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-08-13 16:21:20.264514: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-08-13 16:21:20.266578: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-08-13 16:21:20.266873: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-08-13 16:21:20.268789: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-08-13 16:21:20.269768: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-08-13 16:21:20.273706: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-08-13 16:21:20.274943: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-08-13 16:21:20.275433: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-08-13 16:21:20.300578: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-08-13 16:21:20.302600: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fd480000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-08-13 16:21:20.302646: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-08-13 16:21:20.398547: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1a19cf60 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-08-13 16:21:20.398589: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-08-13 16:21:20.399948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-08-13 16:21:20.400051: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-08-13 16:21:20.400083: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-08-13 16:21:20.400111: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-08-13 16:21:20.400139: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-08-13 16:21:20.400168: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-08-13 16:21:20.400195: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-08-13 16:21:20.400224: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-08-13 16:21:20.402003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-08-13 16:21:20.402080: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-08-13 16:21:20.403171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-08-13 16:21:20.403197: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-08-13 16:21:20.403212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-08-13 16:21:20.405097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 6640 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-08-13 16:21:27.481162: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.470345497131348 time used to load_weights : 0.15419363975524902 0it [00:00, ?it/s]temp/1755094827_129914_1376756114_84dc066d4fe8320f30a7453c61a904de.jpg temp/1755094827_129914_1376756112_4388781093e53389a63342d4714d1e3a.jpg temp/1755094827_129914_1376756108_8b8648c8bc889901d88819be4a7b553b.jpg temp/1755094827_129914_1376756107_02896615351c584261d9b37ef57e96dc.jpg temp/1755094827_129914_1376756106_4f3791a0a9f95547680d4f045e0b7305.jpg temp/1755094827_129914_1376756105_dda4182b3a07d5871f0c980511f1349d.jpg temp/1755094827_129914_1376755917_54e82d08e5dd44c24b8a5c4076c23b56.jpg temp/1755094827_129914_1376755907_63338a4c7754a988afd2a7aa20309f35.jpg temp/1755094827_129914_1376755899_e4825098dcfe91e0346c1da9f9c14d2b.jpg temp/1755094827_129914_1376755884_1fb8324ed261bf175659e8bec4bf715a.jpg temp/1755094827_129914_1376755877_9ddead65505bf499e6c975f9c3ae93a9.jpg temp/1755094827_129914_1376755873_bd5482b1f287d4b8eacfa86197a8e322.jpg temp/1755094827_129914_1376730864_63120729dff5bbafdfddf5f03621be4d.jpg temp/1755094827_129914_1376730861_d134816caa4d94fb0128709a9a24b600.jpg temp/1755094827_129914_1376730751_28f847459ad56d52457f1d9223be3d20.jpg temp/1755094827_129914_1376730748_35d5947f480939813ea9f613d5d7f3cc.jpg temp/1755094827_129914_1376730743_03a4a094ada547a692d6e345864710ac.jpg temp/1755094827_129914_1376730742_f2c7cb108c92ef03c8b9b37c299c9d06.jpg temp/1755094827_129914_1376730741_a4f80750f449743e9c2c54c65818084b.jpg temp/1755094827_129914_1376730740_7092a1bc217a69bdf7805ce3983be674.jpg temp/1755094827_129914_1376730699_4d49fbc98cb2c1e190dcc976388b0fa6.jpg temp/1755094827_129914_1376730697_c0746b5d22e0f906baa8bd5227ea0ae3.jpg temp/1755094827_129914_1376730693_a8337eeb771f7b6b4e2484ae2815b8cb.jpg temp/1755094827_129914_1376730691_dbf6bfbcdd1d464bd6a1187f1616a151.jpg temp/1755094827_129914_1376730690_bc5ce65fb421a2804ceabc001575ede3.jpg temp/1755094827_129914_1376730689_9c3006b90deb8e42788afa344e5f11c4.jpg temp/1755094827_129914_1376730659_5ed7f04e86cbf85efc8ca28011bf2c21.jpg temp/1755094827_129914_1376730657_66ad03a84b9c58aca774951603055eb6.jpg temp/1755094827_129914_1376730655_dbee9960a3fe99d5172489d45c6ae8b8.jpg temp/1755094827_129914_1376730650_15edc12ee90989e61036dac16f0541ff.jpg temp/1755094827_129914_1376730648_a9e592d5ef0baa2105e22372c537f7b7.jpg temp/1755094827_129914_1376730646_e8bd3d79d8b52c04ebb8cbc03b03dc36.jpg temp/1755094827_129914_1376730603_c45106e8a04ab8e90ff0fa9ba7a68447.jpg temp/1755094827_129914_1376730601_de539a067491f8118d318486934faace.jpg temp/1755094827_129914_1376730597_1793f7f554b7a209600e11ff6b56fbeb.jpg temp/1755094827_129914_1376730595_178394db2584236f66cf66896a2f474c.jpg temp/1755094827_129914_1376730594_64d9eb796ff020da23bbe18416216dc9.jpg temp/1755094827_129914_1376730593_ebf9866ed2597a2b5d09f01bdadd5fd4.jpg temp/1755094827_129914_1376730552_85f56edf6f1b0f59b5578eb4ad36ead9.jpg temp/1755094827_129914_1376730550_f331a4581d31c355bf8054e31067d13d.jpg temp/1755094827_129914_1376730546_a07f7be51bbbc87b3a9b64b6b6bcccff.jpg temp/1755094827_129914_1376730543_5dd8cf74179ebdde701f5174645936e1.jpg temp/1755094827_129914_1376730541_f27670a22b9681ea659eb02c5c988134.jpg temp/1755094827_129914_1376730539_b86d9429db6c7bceb8b2208cb42b5f49.jpg temp/1755094827_129914_1376730531_aafd4b2afd13cff069a525c3609671d0.jpg temp/1755094827_129914_1376730527_415ffcd23504bcbbde4e78620ab438e0.jpg temp/1755094827_129914_1376730524_5bae81c4f83dab74bb7c6d7ac1891bc8.jpg temp/1755094827_129914_1376730521_30f96f466869fd8f4108f89ba6350b32.jpg temp/1755094827_129914_1376730519_716043508c9e30deedc1cebd28f0e987.jpg temp/1755094827_129914_1376730516_6b2912a9615c57ff20b8d2618826340b.jpg temp/1755094827_129914_1376730470_eb3443fc06239d3a4adbfa28e8ebd3e5.jpg temp/1755094827_129914_1376730469_1e7370cc9e0f55e67d88db1213890c3d.jpg temp/1755094827_129914_1376730466_aacc9c9a94d1d402495c74ed563211e2.jpg temp/1755094827_129914_1376730464_7c08f2ae1cdbf914620dbc78c42d7d70.jpg temp/1755094827_129914_1376730463_1a4e387d345ac4439425dd541f3d65a7.jpg temp/1755094827_129914_1376730462_b23c522072b67417b585ac8edb0d9ffa.jpg temp/1755094827_129914_1376730416_7c50e88bb4ebab0fee0fcdc2ff2ce424.jpg temp/1755094827_129914_1376730412_a0bb7a7b13fdf6696c66ddc01b0441a2.jpg temp/1755094827_129914_1376730409_a76c570e941fc7003056c8986dac5fff.jpg temp/1755094827_129914_1376730406_87484e13a5f8ddb2ea5b385a2a3b0fa4.jpg temp/1755094827_129914_1376730402_ea8d2779ef42313c3a95f8498f36a78f.jpg temp/1755094827_129914_1376730363_b2f803d3cf8b24c0a51b12ffc98f231e.jpg temp/1755094827_129914_1376730362_d9731246a6fee718ebf3108b198628d1.jpg temp/1755094827_129914_1376730360_7bb45c6db1bd7e3b967fa2012f27050b.jpg temp/1755094827_129914_1376730359_7197a7e1df2cc5e426221974f3671af1.jpg temp/1755094827_129914_1376730357_f6b5b1d004157bb83d1c794dc20d8c49.jpg temp/1755094827_129914_1376730355_8f25bf13f23f5c8136b1b78655f1be2f.jpg temp/1755094827_129914_1376730316_0823d3eddfd37bc3e23286bd8492e6ab.jpg temp/1755094827_129914_1376730315_14229698a6a6679d9894ebaffef660a7.jpg temp/1755094827_129914_1376730313_fbe62cc2550152462cf9d746d571c96e.jpg temp/1755094827_129914_1376730312_1a24f27f94eb62a6ae6031c7feb0b71c.jpg temp/1755094827_129914_1376730311_c12feee63763c98bfc6e3672a30fad04.jpg temp/1755094827_129914_1376730309_69cbd97e5a263e3c894f89820f8c1f8d.jpg temp/1755094827_129914_1376730254_a773a4d06e3e8b60faf81b4c15d6d71e.jpg temp/1755094827_129914_1376730253_b1ee8045d38ac110adcb7796ba2811fb.jpg temp/1755094827_129914_1376730249_22e5f2e56eb07371d8a1dfd5d2d32006.jpg temp/1755094827_129914_1376730248_ce33e63909fb56a498e6ecc2680082dd.jpg temp/1755094827_129914_1376730247_aad699880d009a42afd1fe3bf03d3dfc.jpg temp/1755094827_129914_1376730246_e37fbdebf49dcdf90c518816fa16b22f.jpg temp/1755094827_129914_1376730185_2d035c6fc44044b6ae38174c686268e9.jpg temp/1755094827_129914_1376730158_fddda72dfaf01bff4744e44298770067.jpg temp/1755094827_129914_1376730123_ede9208a5faee221dd4b9ac82c2d767d.jpg temp/1755094827_129914_1376730119_b2503f6e89ced81be12d45c2e6f475ab.jpg temp/1755094827_129914_1376730118_9c84ebb5227205c67d0d15aa99df971e.jpg temp/1755094827_129914_1376730102_ead2b89290b8d41e9e3185cd56aa6dae.jpg temp/1755094827_129914_1376729628_d8d3950a3dc38b3e6a132a7a04026fa6.jpg temp/1755094827_129914_1376729627_fc8d2831e2de4567cdab916763d22d9c.jpg temp/1755094827_129914_1376729625_88fa72db32abdcbeac71dc74bdf794e8.jpg temp/1755094827_129914_1376729604_a72ccdee963a6fda788b1329d74377dd.jpg temp/1755094827_129914_1376729574_99060ebcf43fee8ded8790adea418cbf.jpg temp/1755094827_129914_1376729538_c74b37bb834f6e3c179a6ea1230394f8.jpg temp/1755094827_129914_1376728921_f190045e2118062f14e790a51cfc0938.jpg temp/1755094827_129914_1376728888_76f6d095f0393123dd8eb5c7b6f08fe7.jpg temp/1755094827_129914_1376728856_541a95a34a9674c282159108611fa35a.jpg temp/1755094827_129914_1376728825_8e2457755f4ac0b42cedad770b8ef6f3.jpg temp/1755094827_129914_1376728792_47152f24b9ebc3a90e64838ebfd3fd59.jpg temp/1755094827_129914_1376728753_f9cb9afa88f3db59a1925818671fff44.jpg temp/1755094827_129914_1376728677_c0b88971bbd5edf7c82799c17a2e9385.jpg temp/1755094827_129914_1376728675_de37492baf3cd6c2cba75bf941cf985b.jpg temp/1755094827_129914_1376728671_ceed943a0bfc4f101a2b2ee9990bd28b.jpg temp/1755094827_129914_1376728670_34b7f581160aaa554597f4eb110f7428.jpg temp/1755094827_129914_1376728668_e1dcd8360cc4d7ace80d6e7ce4450893.jpg temp/1755094827_129914_1376728666_051bcd056b256e26636834d4f86a047f.jpg temp/1755094827_129914_1376728548_8ecaf68ed84dcf28fe7f40d3bf342a8a.jpg temp/1755094827_129914_1376728547_c155527f8406411fc73ec10545f3cbe0.jpg temp/1755094827_129914_1376728544_31d89f9ef4dba73a467944d355e50886.jpg temp/1755094827_129914_1376728543_4f28caa7285d1e3e81e9e0ffd3dc71a3.jpg temp/1755094827_129914_1376728541_df6a67dae789ae51a87fd15f53d34a94.jpg temp/1755094827_129914_1376728538_b09df72bf9cb67e857908b0d7bfc6619.jpg temp/1755094827_129914_1376728412_5fda7da1947d6a221eea413a35db40d2.jpg temp/1755094827_129914_1376728410_4abe8427e4872b3315ff02946166198d.jpg temp/1755094827_129914_1376728406_bb372ff050357a2aa27c9423ab71fe16.jpg temp/1755094827_129914_1376728404_d21e2d13655b14085494bf3657f5d415.jpg temp/1755094827_129914_1376728403_b16ae0f3eb8706e1336e0bd0c6cc9a6c.jpg temp/1755094827_129914_1376728402_f7183f8914b0c7b4b77f51de0693f372.jpg 152it [00:00, 1515.48it/s] 199it [00:00, 1558.17it/s]2025-08-13 16:21:31.149004: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-08-13 16:21:31.153076: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-08-13 16:21:31.184886: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-08-13 16:21:31.213603: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-08-13 16:21:31.221568: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1755094827_129914_1376728396_0f75cea8691b7f2eb5e549eca96f2311.jpg temp/1755094827_129914_1376728393_45f7fa9b45e515ef9795247de87aa9e0.jpg temp/1755094827_129914_1376728390_aea63337176a18fa562b4d1d591e2b22.jpg temp/1755094827_129914_1376728387_2896f0ced8563420a529f7a429403752.jpg temp/1755094827_129914_1376728385_f26feca0b97977d1a1418cf68f8c33ac.jpg temp/1755094827_129914_1376728383_c608f87bbc3e98ef65901dbdc5041059.jpg temp/1755094827_129914_1376728031_0dcbfd7d0570d4e0cf8cad989f05ae35.jpg temp/1755094827_129914_1376728030_3a44e235880af7e3ce6d847f8d68df77.jpg temp/1755094827_129914_1376728026_217d7a385a941fdb96d945d78a1e320f.jpg temp/1755094827_129914_1376728025_f701c66e9db085c20529aa038aaad440.jpg temp/1755094827_129914_1376728024_05487359bc873110c35aace4db8dc53d.jpg temp/1755094827_129914_1376728021_35205cfbd68fd3976f653c73898dc3ec.jpg temp/1755094827_129914_1376727963_c604c4496118088032c6b50227c1da42.jpg temp/1755094827_129914_1376727962_7746a53737f39becb3af253fd8af91a5.jpg temp/1755094827_129914_1376727960_4663d000a5733e32147ecdb109668194.jpg temp/1755094827_129914_1376727959_4045e36a9828dff139a505ba434a8189.jpg temp/1755094827_129914_1376727958_eee53c3ee117b01ea42c6d0e83202d75.jpg temp/1755094827_129914_1376727956_d993ac1625e9fb57ac64822030febf95.jpg temp/1755094827_129914_1376727932_a4cc89864eee238d55d2ad2fa2d3114b.jpg temp/1755094827_129914_1376727931_f0a46f846d170f4d5e6b07ef0f1043bb.jpg temp/1755094827_129914_1376727928_441491e071ed451fc89212558860f5b0.jpg temp/1755094827_129914_1376727927_f37c24695a5e680dfe6038267d7acb83.jpg temp/1755094827_129914_1376727926_ecb69d6807c6ea48c5c8390f63dd7c87.jpg temp/1755094827_129914_1376727923_dfafbaeb0ee14d66f5da7aad596ef02d.jpg temp/1755094827_129914_1376727884_2b17eb41eceadd60cd5ec381e3434981.jpg temp/1755094827_129914_1376727883_682999777b28b77ce2c06a0941384c82.jpg temp/1755094827_129914_1376727882_406f0f5f3f9df62d1cb0d0ece7cf0bb8.jpg temp/1755094827_129914_1376727879_7e3581adc46f39d1d228315ca41d8309.jpg temp/1755094827_129914_1376727878_068e80bbc98642ffb451978ae127fea7.jpg temp/1755094827_129914_1376727877_977e0e351135f5079e650892e3ebde53.jpg temp/1755094827_129914_1376727821_b55edf25504fd5e374f29cd7eec1008c.jpg temp/1755094827_129914_1376727820_03e0eb76b4609ddd97350c4ae5946134.jpg temp/1755094827_129914_1376727817_2b381707bde9ade7ecb2a4a44bdc7f07.jpg temp/1755094827_129914_1376727815_ae90e557c75f06bf571587095467c2a1.jpg temp/1755094827_129914_1376727802_7ee892a58d4a8bbb050220ebde265e27.jpg temp/1755094827_129914_1376727782_68769fe795f45b63418c28d31c0c72bb.jpg temp/1755094827_129914_1376727711_7302eabbf43f8bf81019d6636c7d5d06.jpg temp/1755094827_129914_1376727707_60dca36ba8e7070ccbe0489cd5330d7a.jpg temp/1755094827_129914_1376727703_c17bb041ec1ad335cfeec199ffb6e796.jpg temp/1755094827_129914_1376727700_4623024b5c412737af5b4f85ce982361.jpg temp/1755094827_129914_1376727699_170c23df61b1de608b8227183579b216.jpg temp/1755094827_129914_1376727697_64c718dfa301687e593398f4eb90d8a0.jpg temp/1755094827_129914_1376727652_5bc2aeeb8e5dc78d486c5523766f70cd.jpg temp/1755094827_129914_1376727650_51a147cb41c079c7fb1e8a99aafa88d1.jpg temp/1755094827_129914_1376727646_fff2c39026adb361218f398db1b90718.jpg temp/1755094827_129914_1376727645_94f547ac3a57ee4a09662422d09ef906.jpg temp/1755094827_129914_1376727644_1a3a314cd780b60269683b615cbb20f5.jpg temp/1755094827_129914_1376727643_1b3c3880e8937d8c2005338f7590afc5.jpg temp/1755094827_129914_1376727636_341034154d3d10b2ecca5a4075bd8840.jpg temp/1755094827_129914_1376727634_9fe502b8a61f8a15435cf9542762a121.jpg temp/1755094827_129914_1376727631_165c026213042f26e672372761657173.jpg temp/1755094827_129914_1376727628_060a348ca23bef9af9affaede4b813b1.jpg temp/1755094827_129914_1376727625_2c6266725b3449d989d8d18327334ec2.jpg temp/1755094827_129914_1376727620_2088a7f3a5560486791c94a0ee34675f.jpg temp/1755094827_129914_1376727517_a36920dd79a7e9638debcae545e28949.jpg temp/1755094827_129914_1376727515_765cb9e29d5eddb17daa560cfda27418.jpg temp/1755094827_129914_1376727511_ffe91059126cfe7489ae6293dd1035ed.jpg temp/1755094827_129914_1376727508_398062349169c7c8fd85646aa8a03d57.jpg temp/1755094827_129914_1376727507_28388f0ad077a239bd1835b325a9dce3.jpg temp/1755094827_129914_1376727505_417ce3b7d9130039eee7778ad2667c37.jpg temp/1755094827_129914_1376727498_2657196d68dcd5d29861c5df01e28630.jpg temp/1755094827_129914_1376727495_af577dfb194a0fdaaf2c4afd4784ed3a.jpg temp/1755094827_129914_1376727491_f2a5c41c5f376449a49443429b842748.jpg temp/1755094827_129914_1376727488_902dc383bce39b6b0054faa3b006354a.jpg temp/1755094827_129914_1376727486_62fb0b38f987334c3503160974abfa6a.jpg temp/1755094827_129914_1376727484_be353a445cfa1baae2689facac6f68cc.jpg temp/1755094827_129914_1376727442_dd6fea42d6019b894e5653b8454a7f84.jpg temp/1755094827_129914_1376727440_502a40ce954c5812e900e16be88928c8.jpg temp/1755094827_129914_1376727437_29779d73430e6f2eab5bd9d44a92f7e5.jpg temp/1755094827_129914_1376727436_2d1150d9280dc12edaf2c453f3ba599c.jpg temp/1755094827_129914_1376727433_32036f4a89fe3e4aa711013706166633.jpg temp/1755094827_129914_1376727430_9fd2f32c188f013ba2f97b9d50d7cab3.jpg temp/1755094827_129914_1376726303_da39ba21e1b26f92fac2a29ca1fcb587.jpg temp/1755094827_129914_1376726257_4be1e71ed64fd7bb2512afc587484b31.jpg temp/1755094827_129914_1376726223_0153259adf17f1e0d6cb76d759269033.jpg temp/1755094827_129914_1376726192_886e1a217659287beb59adb4f81aee40.jpg temp/1755094827_129914_1376726158_3323711424e5f79409255e4adaba6656.jpg temp/1755094827_129914_1376726122_0326078465a3eb7498a3fb252018dab7.jpg temp/1755094827_129914_1376725999_d75635fd1afc5b04f1828a0c1a269477.jpg temp/1755094827_129914_1376725995_aa16279f8f07af632099b64f6130c1cb.jpg temp/1755094827_129914_1376725993_59c860b60500cda9dc4795f2536b992d.jpg temp/1755094827_129914_1376725990_689181d1743f6be4bb226089ae2688cc.jpg temp/1755094827_129914_1376725987_7c9bc175d95e3230b937f91d998890bc.jpg temp/1755094827_129914_1376725982_2f25ed3dc7ba456dda4cbc4cc262b23b.jpg Found 199 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1376756114, 1376756112, 1376756108, 1376756107, 1376756106, 1376756105, 1376755917, 1376755907, 1376755899, 1376755884, 1376755877, 1376755873, 1376730864, 1376730861, 1376730751, 1376730748, 1376730743, 1376730742, 1376730741, 1376730740, 1376730699, 1376730697, 1376730693, 1376730691, 1376730690, 1376730689, 1376730659, 1376730657, 1376730655, 1376730650, 1376730648, 1376730646, 1376730603, 1376730601, 1376730597, 1376730595, 1376730594, 1376730593, 1376730552, 1376730550, 1376730546, 1376730543, 1376730541, 1376730539, 1376730531, 1376730527, 1376730524, 1376730521, 1376730519, 1376730516, 1376730470, 1376730469, 1376730466, 1376730464, 1376730463, 1376730462, 1376730416, 1376730412, 1376730409, 1376730406, 1376730402, 1376730363, 1376730362, 1376730360, 1376730359, 1376730357, 1376730355, 1376730316, 1376730315, 1376730313, 1376730312, 1376730311, 1376730309, 1376730254, 1376730253, 1376730249, 1376730248, 1376730247, 1376730246, 1376730185, 1376730158, 1376730123, 1376730119, 1376730118, 1376730102, 1376729628, 1376729627, 1376729625, 1376729604, 1376729574, 1376729538, 1376728921, 1376728888, 1376728856, 1376728825, 1376728792, 1376728753, 1376728677, 1376728675, 1376728671, 1376728670, 1376728668, 1376728666, 1376728548, 1376728547, 1376728544, 1376728543, 1376728541, 1376728538, 1376728412, 1376728410, 1376728406, 1376728404, 1376728403, 1376728402, 1376728396, 1376728393, 1376728390, 1376728387, 1376728385, 1376728383, 1376728031, 1376728030, 1376728026, 1376728025, 1376728024, 1376728021, 1376727963, 1376727962, 1376727960, 1376727959, 1376727958, 1376727956, 1376727932, 1376727931, 1376727928, 1376727927, 1376727926, 1376727923, 1376727884, 1376727883, 1376727882, 1376727879, 1376727878, 1376727877, 1376727821, 1376727820, 1376727817, 1376727815, 1376727802, 1376727782, 1376727711, 1376727707, 1376727703, 1376727700, 1376727699, 1376727697, 1376727652, 1376727650, 1376727646, 1376727645, 1376727644, 1376727643, 1376727636, 1376727634, 1376727631, 1376727628, 1376727625, 1376727620, 1376727517, 1376727515, 1376727511, 1376727508, 1376727507, 1376727505, 1376727498, 1376727495, 1376727491, 1376727488, 1376727486, 1376727484, 1376727442, 1376727440, 1376727437, 1376727436, 1376727433, 1376727430, 1376726303, 1376726257, 1376726223, 1376726192, 1376726158, 1376726122, 1376725999, 1376725995, 1376725993, 1376725990, 1376725987, 1376725982] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 199 time used for this insertion : 0.08140277862548828 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 13.795676708221436 time spend to save output : 0.08541607856750488 total time spend for step 0 : 13.88109278678894 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 13.99user 3.95system 1:08.53elapsed 26%CPU (0avgtext+0avgdata 1800124maxresident)k 256inputs+355408outputs (3major+598059minor)pagefaults 0swaps