python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2868126 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 1640>200 please execute split_portfolio.py -i 27896762 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['3957315'] with mtr_portfolio_ids : ['27896762'] and first list_photo_ids : [] new path : /proc/2868126/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 39.34283423423767 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Sat Oct 18 11:51:23 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2248 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2248 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2248 wait 20 seconds inside check gpu memory 2025-10-18 11:52:29.326975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-10-18 11:52:29.334729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-10-18 11:52:29.334936: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-18 11:52:29.336525: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-18 11:52:29.338050: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-10-18 11:52:29.338328: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-10-18 11:52:29.340190: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-10-18 11:52:29.341121: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-10-18 11:52:29.344777: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-18 11:52:29.345749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-10-18 11:52:29.346073: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-10-18 11:52:29.370471: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493010000 Hz 2025-10-18 11:52:29.372046: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f5854000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-10-18 11:52:29.372072: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-10-18 11:52:29.586407: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3e8c3e90 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-10-18 11:52:29.586442: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-10-18 11:52:29.587415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-10-18 11:52:29.587500: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-18 11:52:29.587533: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-18 11:52:29.587564: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-10-18 11:52:29.587594: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-10-18 11:52:29.587624: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-10-18 11:52:29.587654: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-10-18 11:52:29.587684: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-18 11:52:29.589021: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-10-18 11:52:29.589087: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-10-18 11:52:29.589920: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-10-18 11:52:29.589943: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-10-18 11:52:29.589956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-10-18 11:52:29.591373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3441 max_wait_temp : 4 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.132793664932251 time used to load_weights : 0.19958281517028809 0it [00:00, ?it/s] 112it [00:00, 1080.27it/s]temp/1760781044_2868126_1389936916_8b7ff3f7d47256e6c632990448b39065.jpg temp/1760781044_2868126_1389936872_cb257157403d74aeffb5ec3960acb9b6.jpg temp/1760781044_2868126_1389936805_a487fa8ff8aada3160d6dcd706f1a378.jpg temp/1760781044_2868126_1389936762_6d4aefde1e00e42d39ce8b5b191edbc6.jpg temp/1760781044_2868126_1389936687_634cb846aded129b42340317c192a965.jpg temp/1760781044_2868126_1389936671_ddde512ae1a01dedc5bfaa3eb2582830.jpg temp/1760781044_2868126_1389936636_a2da96554cba40eefd325797efb0c7cb.jpg temp/1760781044_2868126_1389936635_85b6a8f3b336e7193ce2b8074e5056c6.jpg temp/1760781044_2868126_1389936605_d6cdef2c7259b4603ecb3d9ee050217c.jpg temp/1760781044_2868126_1389936564_8172f6a464fef29699465b0293c58996.jpg temp/1760781044_2868126_1389936558_ed117ef8f99901807d4ec2766c1a1076.jpg temp/1760781044_2868126_1389936555_92e4bbaf790868e248700e6bfc945822.jpg temp/1760781044_2868126_1389936554_37b2f7a61bd8d3de92da298715aecd01.jpg temp/1760781044_2868126_1389936550_a84ed5b629358114563b1da1e70cba20.jpg temp/1760781044_2868126_1389936542_9eafb14c42402551f12471fe966fcf41.jpg temp/1760781044_2868126_1389936532_c446796e19ab7eef700027fdc8f5ddf2.jpg temp/1760781044_2868126_1389936531_45a8872b264f3ba071b26604c8f5208d.jpg temp/1760781044_2868126_1389936487_536b4c93debd04ac04a87025a3d76f65.jpg temp/1760781044_2868126_1389936460_9bb4b83e1134e6a6c567e7cb234f34eb.jpg temp/1760781044_2868126_1389936457_48a0a6db0434748c35578a7cff679db6.jpg temp/1760781044_2868126_1389936441_114fc08882e28eabc4ad88fad3d53b22.jpg temp/1760781044_2868126_1389936440_8676a7638dc26f3957ea0a8784b851c8.jpg temp/1760781044_2868126_1389936439_04072f62c7ce867c9f92cbdd976b307a.jpg temp/1760781044_2868126_1389936435_fbb48329e87e667c7b86d1dbe8a706d1.jpg temp/1760781044_2868126_1389936409_4c7f54b98a2ed2759350161bd0f844bc.jpg temp/1760781044_2868126_1389936377_faade2a5e56ade3570c8f95067c97249.jpg temp/1760781044_2868126_1389936376_460d4e445d189bcde63812cd9abc6457.jpg temp/1760781044_2868126_1389936368_e399f2bfd916e8bd3fd63d21edc549c8.jpg temp/1760781044_2868126_1389936350_59d21e7b81b9683110be02542c11279f.jpg temp/1760781044_2868126_1389936345_08216942ceb16493301e5275829df7d6.jpg temp/1760781044_2868126_1389936332_dc318a61086f1743f347675045e30043.jpg temp/1760781044_2868126_1389936331_fcadd03a5f3537fb38ea990d88e5a62d.jpg temp/1760781044_2868126_1389936328_01353845c9c5824f08d0a4876939c496.jpg temp/1760781044_2868126_1389936304_24567ab942f744cf40eb96e055787e55.jpg temp/1760781044_2868126_1389936296_b9ba6e50541b727deda659db9a556896.jpg temp/1760781044_2868126_1389936290_f2200e52a6098a9a188be9e87f74fbd0.jpg temp/1760781044_2868126_1389936263_1de5d1243fc02811353248cc6a46b46e.jpg temp/1760781044_2868126_1389936260_175fb3a84a0ed44b67f5557d470023e9.jpg temp/1760781044_2868126_1389936246_de1ed9f96b84073d664200e0537b3dd1.jpg temp/1760781044_2868126_1389936234_d8ce6bf62711170b16cbb39e3ed021e5.jpg temp/1760781044_2868126_1389936201_ce46bb25cccc2adaba1e2a1b6ed1121a.jpg temp/1760781044_2868126_1389936143_f7e599a63e63d66b34e51bcb155cd39f.jpg temp/1760781044_2868126_1389936131_8f232a667d80b2086128c4b51ea070f0.jpg temp/1760781044_2868126_1389936100_b9c90c1f65ced998c78683878d35fb39.jpg temp/1760781044_2868126_1389936055_17893bc7b856b178f6825990ff793a7c.jpg temp/1760781044_2868126_1389936022_61fde0ddb06be2b42a09b72455e632d6.jpg temp/1760781044_2868126_1389935991_8765d2e0c7eb672c5510dd3b5e20e606.jpg temp/1760781044_2868126_1389935982_c2b0e3b7e95bdbde9a33c60af8f6ef46.jpg temp/1760781044_2868126_1389935960_412ebda7a79da386649ec3fea5bc9ab7.jpg temp/1760781044_2868126_1389935957_6e2c818fece4b4d3c16138bd615f388a.jpg temp/1760781044_2868126_1389935943_c63a05ab8cb1e4deb7e95a2f351a9fc9.jpg temp/1760781044_2868126_1389935889_dd8d77d7a3794626b1533800b2126940.jpg temp/1760781044_2868126_1389935864_80fc7ecb09af5a89aa8a0ddac49c1f86.jpg temp/1760781044_2868126_1389935843_7e00253d55225fac167a9d67b0b59f97.jpg temp/1760781044_2868126_1389935842_ef7d03233ee84b117956375e4f0855c6.jpg temp/1760781044_2868126_1389935804_085f10bc018444fa7bba84ae820f5815.jpg temp/1760781044_2868126_1389935802_81306ff3005218684c323247813582af.jpg temp/1760781044_2868126_1389935784_c3f0f7a4c32d0e57c23fc37f44fb7113.jpg temp/1760781044_2868126_1389935763_32b4facea18af136c23bd7f1c4b4dbcc.jpg temp/1760781044_2868126_1389935746_0cd9f0babda7acaab95f2fea45d5de66.jpg temp/1760781044_2868126_1389935744_c52a402edf9601522733decc9ad7ea11.jpg temp/1760781044_2868126_1389935705_29cd58f4417444463a2fc47939b3fd7e.jpg temp/1760781044_2868126_1389935704_4204670aaf520f669d3002a661ba91a5.jpg temp/1760781044_2868126_1389935683_c51aeafb40a074c4aac6425ec9c920dd.jpg temp/1760781044_2868126_1389935659_c96778e9affab2e3e9a9623349259877.jpg temp/1760781044_2868126_1389935657_f15028ae5cd8ef2ea06cbe3d1715bfed.jpg temp/1760781044_2868126_1389935639_ef0e9fefe9f296e5c038b7ad4abcf3b0.jpg temp/1760781044_2868126_1389935638_46932c1d875025c84d8a53e9c88a7571.jpg temp/1760781044_2868126_1389935601_757ca665ce5751a67078ddf7cf48486d.jpg temp/1760781044_2868126_1389935590_29d62c482fb25c4e237a38b3abd96863.jpg temp/1760781044_2868126_1389935565_f7855468afa0918a5bdcb681dd4be0d8.jpg temp/1760781044_2868126_1389935563_dd6721a70c738181794bb0523990a793.jpg temp/1760781044_2868126_1389935550_f0d987099160565c1ad7899cfc68c984.jpg temp/1760781044_2868126_1389935536_e36747cfed592390a074f7a42df7c1f3.jpg temp/1760781044_2868126_1389935534_73154faff4d6c14a1bda32a538496b81.jpg temp/1760781044_2868126_1389935510_fa47d29238697c2fa7a73a754b031956.jpg temp/1760781044_2868126_1389935488_da3c374de13de9449bbb397591819bf0.jpg temp/1760781044_2868126_1389935483_8d36942ea59136f93cea084a2e461f85.jpg temp/1760781044_2868126_1389935458_2d80365bddb0b35861a01c62cc8ddb4f.jpg temp/1760781044_2868126_1389935455_71a9a87cfe6e17462425a8ad1431223e.jpg temp/1760781044_2868126_1389935447_4fc921d8335b5cd03c066bfb5cf5e29d.jpg temp/1760781044_2868126_1389935432_ca0e230f435c3df569ffc47151ad7508.jpg temp/1760781044_2868126_1389935427_73178bd54b5981f512a05cefa035d2ad.jpg temp/1760781044_2868126_1389935425_81caa18f253163804758b96938eb5210.jpg temp/1760781044_2868126_1389935364_45182715a7b37940f6562e661b50a531.jpg temp/1760781044_2868126_1389935327_97ebb240dd01f350817d35d6b8d40edd.jpg temp/1760781044_2868126_1389935277_66b3ec313938813b8fa16fc94fff4c37.jpg temp/1760781044_2868126_1389935241_29ef45875a4433539845c3e85b7cd22f.jpg temp/1760781044_2868126_1389935240_24b8d56887e24bef1b60c19380db1075.jpg temp/1760781044_2868126_1389935191_3c1d97d249443865d82d851a35f59d58.jpg temp/1760781044_2868126_1389935184_8cae97772389833a6c22f24edd3a40ea.jpg temp/1760781044_2868126_1389935155_1e347fbc0b58d1e1aef3489a33729202.jpg temp/1760781044_2868126_1389935151_87e4b4c19dd0d959bca2b9953cbe2c49.jpg temp/1760781044_2868126_1389935095_5d4584220348c9473c03dbab19ce1f0e.jpg temp/1760781044_2868126_1389935078_0772337a7f219cb119021760b33fff79.jpg temp/1760781044_2868126_1389935063_2c31f450ca5719a951935439897c558b.jpg temp/1760781044_2868126_1389934993_c190a4bb621f343ae940df3debd80c6b.jpg temp/1760781044_2868126_1389934979_f5ad2bffefb60d4c99181134975c99dd.jpg temp/1760781044_2868126_1389934944_7db69b596d33c7c4adf9286584af0e94.jpg temp/1760781044_2868126_1389934907_8ece53062b974d7d25c9e387f8521399.jpg temp/1760781044_2868126_1389934885_c7cab8aff3251cb0a298f5439147ed19.jpg temp/1760781044_2868126_1389934860_e55703eaf9c7845186f751d366cd87f2.jpg temp/1760781044_2868126_1389934833_c7f0a9c6400a8e287f38bb8a86ae2a34.jpg temp/1760781044_2868126_1389934812_dd363ca2f565590afdddcd5c082c4274.jpg temp/1760781044_2868126_1389934793_5ecd7ff88282f43ee7b10f1b9bfbd579.jpg temp/1760781044_2868126_1389934775_2cab9bb8065c34e7e6bed79827f6ef68.jpg temp/1760781044_2868126_1389934774_d63f8cff18c47a90cd824b158a2d8dfb.jpg temp/1760781044_2868126_1389934729_ce72d23d4bbd26c8a832f5b10740c761.jpg temp/1760781044_2868126_1389934688_595b62626ca6a515899c071e7c971166.jpg temp/1760781044_2868126_1389934687_d255d543686cbf998ae7a92675b356b4.jpg temp/1760781044_2868126_1389934607_50a803f481ce06f8be7a35aedd191209.jpg temp/1760781044_2868126_1389934606_f47322043a14a62bf44ca642f1aef763.jpg temp/1760781044_2868126_1389934522_416fc50fc22a12af127a0faad019ae13.jpg 200it [00:00, 1002.96it/s]2025-10-18 11:52:39.476365: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-10-18 11:52:39.600305: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-10-18 11:52:39.674148: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-10-18 11:52:39.676200: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-10-18 11:52:39.677412: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1760781044_2868126_1389934516_e1dd83c20dba12a19382c00640af698c.jpg temp/1760781044_2868126_1389934472_018da2b13586161f0923c01d06b12742.jpg temp/1760781044_2868126_1389934471_87839f3e7c7f3b64b35ef83f9305221e.jpg temp/1760781044_2868126_1389934451_9927dda165e9620ba3085f3a64856dcc.jpg temp/1760781044_2868126_1389934411_f6972667939ff9c85b88a3d05f9e87ba.jpg temp/1760781044_2868126_1389934404_5f93c2d81b55de96d5cf216eaa530640.jpg temp/1760781044_2868126_1389934388_c5f4c8b4df7209d5be8572bfe82b319a.jpg temp/1760781044_2868126_1389934379_2d2caecf6421fba9609f04a4aae4e908.jpg temp/1760781044_2868126_1389934308_d0b61d49f4efb7fcda6a364efdbef664.jpg temp/1760781044_2868126_1389934306_9b3de88033faba9ba0247f649d2ff1be.jpg temp/1760781044_2868126_1389934289_7d08ea60325778d0f8f74b1a9c148e14.jpg temp/1760781044_2868126_1389934256_366765135c2455d6e0ea2c0cba85c291.jpg temp/1760781044_2868126_1389934223_0f9a7410322e54c4ef40786c451f6ed0.jpg temp/1760781044_2868126_1389934208_74371d7e01ad69337ad898536d440e7f.jpg temp/1760781044_2868126_1389934129_e8593655ca12350f1c439aa3a9906e46.jpg temp/1760781044_2868126_1389934090_37473542cc86b16d7c8dd7f2bc7d3475.jpg temp/1760781044_2868126_1389934029_76995c104ebf5ae890f672521389e27d.jpg temp/1760781044_2868126_1389934008_56cad4d9eb4a66b2a339154316b31252.jpg temp/1760781044_2868126_1389934007_cb0f3fad58765aa1873f48567964db8e.jpg temp/1760781044_2868126_1389933970_fd1125db7d1b6628b8dee3bd488227be.jpg temp/1760781044_2868126_1389933936_b33c8d43a6d5c5397d0f0beb18c4863c.jpg temp/1760781044_2868126_1389933924_e997d7b7ea7d6a656912c33a6b4f5a27.jpg temp/1760781044_2868126_1389933907_a2364a80ced71e3165d36e87d15fc53e.jpg temp/1760781044_2868126_1389933896_fec032094fc982bcbafabc162592a3c1.jpg temp/1760781044_2868126_1389933895_553fc5f73eba5dec17fbee8c83d040fa.jpg temp/1760781044_2868126_1389933873_d60add317d58cb86f8d5b071d54e6f4f.jpg temp/1760781044_2868126_1389933839_02529a1f000ff87878babf8af6d9a26e.jpg temp/1760781044_2868126_1389933813_3103a21a6806e789f28ebe0de6c3dd10.jpg temp/1760781044_2868126_1389933800_6d25534d18677ebae5acb7eb2722ed9f.jpg temp/1760781044_2868126_1389933798_4a6c79179b1caacc6b188866940fd9db.jpg temp/1760781044_2868126_1389933787_6e3d5ca32d03cfb2d4521d2707352ee4.jpg temp/1760781044_2868126_1389933767_44b7d51ad6eb3be1000145a97e9b7153.jpg temp/1760781044_2868126_1389933751_d158cfa84647950886a75d91e0e061f9.jpg temp/1760781044_2868126_1389933723_97a1f0e83a28ddf206b4ba04d5ec98bb.jpg temp/1760781044_2868126_1389933710_55a39a7fe46b2bd6cf41437a378ab8ce.jpg temp/1760781044_2868126_1389933689_bdb1033bbde6956535cb6bfb86590742.jpg temp/1760781044_2868126_1389933659_951cfd82a2f9b61736ebb0ec76ea5882.jpg temp/1760781044_2868126_1389933624_0195c68529049c039dbd1824ffc77734.jpg temp/1760781044_2868126_1389933619_aa346d0904d948c4ab94dbb6fe3360a4.jpg temp/1760781044_2868126_1389933582_942ffee07d1102a48c802f79044e3a3a.jpg temp/1760781044_2868126_1389933574_e02e622c1959ad18f322aa7018a7cfaf.jpg temp/1760781044_2868126_1389933548_475d59fe6b1bcebbeac29a5166f0af3f.jpg temp/1760781044_2868126_1389933531_5c224e42a5c3eb63ec72b9c1691b459e.jpg temp/1760781044_2868126_1389933497_c6da884dcd08eff9293f947728835cf9.jpg temp/1760781044_2868126_1389933479_fb0ba2753ca6aff69f2f71623c00fb11.jpg temp/1760781044_2868126_1389933445_2fc3540a3dfe88cc388f61d7d63b409b.jpg temp/1760781044_2868126_1389933391_cb8eae2f12facb446542ed1916a9c402.jpg temp/1760781044_2868126_1389933386_ee1adb99f54c526ed196d36a658b1260.jpg temp/1760781044_2868126_1389933360_fe7118362af9aa54e399bc5bf9daf2a9.jpg temp/1760781044_2868126_1389933313_cc08ef88ec5404bd27f88f97e0af5ac1.jpg temp/1760781044_2868126_1389933312_84897a7c7b2bb2ead4e84ffb5a02703e.jpg temp/1760781044_2868126_1389933281_232a3f1854236ff2711e8e95beda6f75.jpg temp/1760781044_2868126_1389933236_a13f46d18a455a52d6e46d486eaeb44b.jpg temp/1760781044_2868126_1389933231_19ec9a52a24f3f718ec6d9993f346db0.jpg temp/1760781044_2868126_1389933179_1b4b93412077fb8a53470b14aa87ac81.jpg temp/1760781044_2868126_1389933172_9e93665c961334fe79cd4465300a1181.jpg temp/1760781044_2868126_1389933154_5dd0f6170f98a5674e8a4897360a5f5e.jpg temp/1760781044_2868126_1389933100_36b16233f124fcfdae25aab44475c54f.jpg temp/1760781044_2868126_1389933072_d2eda1850f0fe4ea58c64e4b962a57ed.jpg temp/1760781044_2868126_1389933000_7c60940652fb48aa8ec4989d935e94e5.jpg temp/1760781044_2868126_1389932972_bcddeca6ed2fac281e69abe85b7ba79b.jpg temp/1760781044_2868126_1389932898_6d4465a7d7bade459facd34fa2b302c8.jpg temp/1760781044_2868126_1389932879_7fa3b153fde299dab4a783c25a7511c6.jpg temp/1760781044_2868126_1389932820_9a92cf5ae800b1deba712cefabfc216b.jpg temp/1760781044_2868126_1389932818_410dc7fa737c2cdb3988e6389a24a15d.jpg temp/1760781044_2868126_1389932809_b790ee240af089473d9907aebc4beed5.jpg temp/1760781044_2868126_1389932753_43294cda053d746e3e9bdf252ffe6514.jpg temp/1760781044_2868126_1389932730_32016309cff3395badcfa81783fd8cec.jpg temp/1760781044_2868126_1389932667_712b63a1fd231c1df13906c6f6c65587.jpg temp/1760781044_2868126_1389932651_153d5293ce75106690cc6b50bb74f4e3.jpg temp/1760781044_2868126_1389932618_76727a2348e6d099e4ae0118ef0923fd.jpg temp/1760781044_2868126_1389932596_5c21d0b180bf7af1cf714a6f29719fcd.jpg temp/1760781044_2868126_1389932550_a118b86ed5d2bb3328cdf5e0847e04a7.jpg temp/1760781044_2868126_1389932526_77a3b2d26fab0f0ba105ba2a13feef15.jpg temp/1760781044_2868126_1389932449_7b2f24672b83fabbe148471bd14bef68.jpg temp/1760781044_2868126_1389932448_8291e4618601e36e6a6b1ff98d670511.jpg temp/1760781044_2868126_1389932397_eb4389369586b6a402bdd0965016f4e7.jpg temp/1760781044_2868126_1389932387_aa812ab40d3b9723197729c2f2074c6b.jpg temp/1760781044_2868126_1389932321_fff5c10456fd9cd618368754743080a1.jpg temp/1760781044_2868126_1389932307_ad868fe4082beddada5695653eb03334.jpg temp/1760781044_2868126_1389932305_41159d8f1f9e5ce759ce333fc626fdb2.jpg temp/1760781044_2868126_1389932243_ca97fabd393b3c579ce6f4f9afb5c16f.jpg temp/1760781044_2868126_1389932234_ea7adb309676649434ea958a648df260.jpg temp/1760781044_2868126_1389932229_fceb0844a9f6d80ac8c8330230ae90df.jpg temp/1760781044_2868126_1389932226_6732764fb6b44a4a5aeffe9caccf2171.jpg temp/1760781044_2868126_1389932176_ae96419c55e64b34cecc44c153eda420.jpg temp/1760781044_2868126_1389932166_c0a7115d422699df230bae5973e54435.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2339, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2533, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1389936916, 1389936872, 1389936805, 1389936762, 1389936687, 1389936671, 1389936636, 1389936635, 1389936605, 1389936564, 1389936558, 1389936555, 1389936554, 1389936550, 1389936542, 1389936532, 1389936531, 1389936487, 1389936460, 1389936457, 1389936441, 1389936440, 1389936439, 1389936435, 1389936409, 1389936377, 1389936376, 1389936368, 1389936350, 1389936345, 1389936332, 1389936331, 1389936328, 1389936304, 1389936296, 1389936290, 1389936263, 1389936260, 1389936246, 1389936234, 1389936201, 1389936143, 1389936131, 1389936100, 1389936055, 1389936022, 1389935991, 1389935982, 1389935960, 1389935957, 1389935943, 1389935889, 1389935864, 1389935843, 1389935842, 1389935804, 1389935802, 1389935784, 1389935763, 1389935746, 1389935744, 1389935705, 1389935704, 1389935683, 1389935659, 1389935657, 1389935639, 1389935638, 1389935601, 1389935590, 1389935565, 1389935563, 1389935550, 1389935536, 1389935534, 1389935510, 1389935488, 1389935483, 1389935458, 1389935455, 1389935447, 1389935432, 1389935427, 1389935425, 1389935364, 1389935327, 1389935277, 1389935241, 1389935240, 1389935191, 1389935184, 1389935155, 1389935151, 1389935095, 1389935078, 1389935063, 1389934993, 1389934979, 1389934944, 1389934907, 1389934885, 1389934860, 1389934833, 1389934812, 1389934793, 1389934775, 1389934774, 1389934729, 1389934688, 1389934687, 1389934607, 1389934606, 1389934522, 1389934516, 1389934472, 1389934471, 1389934451, 1389934411, 1389934404, 1389934388, 1389934379, 1389934308, 1389934306, 1389934289, 1389934256, 1389934223, 1389934208, 1389934129, 1389934090, 1389934029, 1389934008, 1389934007, 1389933970, 1389933936, 1389933924, 1389933907, 1389933896, 1389933895, 1389933873, 1389933839, 1389933813, 1389933800, 1389933798, 1389933787, 1389933767, 1389933751, 1389933723, 1389933710, 1389933689, 1389933659, 1389933624, 1389933619, 1389933582, 1389933574, 1389933548, 1389933531, 1389933497, 1389933479, 1389933445, 1389933391, 1389933386, 1389933360, 1389933313, 1389933312, 1389933281, 1389933236, 1389933231, 1389933179, 1389933172, 1389933154, 1389933100, 1389933072, 1389933000, 1389932972, 1389932898, 1389932879, 1389932820, 1389932818, 1389932809, 1389932753, 1389932730, 1389932667, 1389932651, 1389932618, 1389932596, 1389932550, 1389932526, 1389932449, 1389932448, 1389932397, 1389932387, 1389932321, 1389932307, 1389932305, 1389932243, 1389932234, 1389932229, 1389932226, 1389932176, 1389932166] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 59.524524450302124 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 76.04982113838196 time spend to save output : 59.529361724853516 total time spend for step 0 : 135.57918286323547 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 12.98user 3.78system 3:06.77elapsed 8%CPU (0avgtext+0avgdata 1853924maxresident)k 424312inputs+706584outputs (116major+497386minor)pagefaults 0swaps